KR20210062988A - Multilateral participation remote collaboration system based on Augmented reality sharing and method thereof - Google Patents

Abstract

A multi-participation remote collaboration system based on augmented reality sharing and a method thereof are disclosed. According to one embodiment, in the multi-participation remote collaboration system and the method thereof, in an augmented reality-based remote guidance environment in which a procedure to operate devices on the work site is provided as an augmented reality manual to be worked, when a situation occurs where workers need help from experts, the augmented reality manual is shared with experts to receive the technology support in real time, and the technology support can be jointly obtained through multi-participation remote collaboration.

Description

Multilateral participation remote collaboration system based on Augmented reality sharing and method thereof}

The present invention relates to image analysis and remote control technology.

Many years of experience are needed to control, remote repair, and maintain a device, and it takes a long time for even an experienced person to learn it when introducing a new device. In addition, no matter how skilled the user is, he or she cannot be familiar with all the work procedures, and it is difficult to predict when and which work demands will occur. Accordingly, workers have always carried a manual for work, and have performed work.

These problems are not limited only to industrial sites. For example, if a refrigerator breaks down at home, consumers are often unable to fix it themselves, so they often have to call a separate professional to repair it. In this case, it is inefficient because it takes a lot of time and money to solve the problem.

According to an embodiment, augmented reality sharing-based multi-participation remote that enables users to work easily and quickly even with devices having a complex structure or even for non-skilled workers, and to receive technical support through remote collaboration of experts when they need help from experts We propose a collaboration system and its method.

Augmented reality sharing-based multi-participation remote collaboration system according to an embodiment acquires an image captured in real-time of a worker's on-site situation, recognizes a work object from the obtained captured image, and then adds augmented reality to the captured image including the recognized work object. A worker terminal that combines and outputs the manual, at least one expert terminal that shares the augmented reality manual with the worker terminal and other expert terminals, and provides technical support for the worker's work based on the augmented reality manual, the worker terminal and It provides a common augmented reality manual to each expert terminal and includes a remote server that remotely controls the operator terminal according to the received command when receiving an expert command for technical support from the expert terminal.

The remote server includes a state tracking unit that tracks the usage location of the worker terminal and each expert terminal, receives the worker's usage location information from the worker terminal, delivers it to each expert terminal, and receives the expert's usage location from each expert terminal Technical support to the operator's terminal remotely from the expert by requesting the expert's terminal and receiving the expert's command when necessary, upon the request of the operator, or by identifying whether the operator requires additional technical support by the expert may include technical support. Here, the use position of the operator terminal may be the current working stage of the operator in the augmented reality manual, and the use position for each expert terminal may be an expert command input from the expert.

The remote server includes a monitoring unit that monitors the operator terminal screen of the operator to determine the operation state of the augmented reality manual, a client management unit that receives real-time state information about the operator terminal to determine the operator terminal state, and the operator terminal usage information of the operator It further comprises a data analysis unit to analyze the worker status, and the technical support unit at least one of the augmented reality manual operation status identified through the monitoring unit, the worker terminal status identified through the client management unit, and the worker status identified through the data analysis unit can be used to determine whether experts provide technical support and select experts to request technical support.

The technical support department includes a work step approach technology that approaches a predetermined work step in the augmented reality manual, a synchronization technology that synchronizes the augmented reality manual between the worker terminal and the expert terminal, an initialization technology that initializes the augmented reality manual, and an annotation tool. Alternatively, through a tool including a direction tool, at least one of a tool using technology supporting technology and a technology supporting technology through video, chatting, or voice may be supported. Here, the predetermined work step approach is based on the current work step, moving the work step in the reverse direction, moving the work step in the forward direction, repeating the current work step, moving to the first work step, or moving to another augmented reality manual. may include

The remote server may include a collaboration support unit that relays technical support collaboration between a plurality of experts through a plurality of expert terminals.

The collaboration support unit may provide a remote conference function in which a plurality of experts can participate simultaneously, and may receive audio along with an augmented reality image from each expert terminal during a remote conference and deliver it to a worker terminal.

The collaboration support unit may define expert command priorities in advance, and when expert commands are received together from a plurality of expert terminals during collaboration between expert terminals, the expert commands may be sequentially processed according to the predefined command priorities.

The collaboration support unit may mediate to consult between the corresponding expert terminals when the expert commands are received together from a plurality of expert terminals.

The remote server may share the augmented reality guide screen including the augmented reality manual of the worker terminal in the reverse direction to the expert terminal of the site that is operating normally, and then remotely control the worker terminal according to the expert command of the expert terminal.

According to the augmented reality sharing-based multi-participation remote collaboration system and method according to an embodiment, the worker receives the help of an expert in the augmented reality-based remote guidance environment in which the procedure for operating the device at the work site is provided as an augmented reality manual. If a situation arises that needs to be received, technical support services can be provided in real time by sharing the augmented reality manual with experts.

In the augmented reality-based remote guidance environment, multi-participation of experts can provide technical support by maximizing augmented reality functions beyond the existing simple remote support. For example, since a number of experts can share an augmented reality manual based on augmented reality and provide technical support to workers using the shared augmented reality manual, the effect of using augmented reality can be maximized.

When the augmented reality manual is shared, workers do not need to receive separate explanations from experts, and experts can more easily identify the wrong part and provide technical support accurately using the augmented reality manual. The operator can work easily and quickly even with the complex structure of the device or even if the operator is not skilled.

1 is a view showing an operator terminal screen in an augmented reality-based remote guidance environment according to an embodiment of the present invention;
2 is a diagram showing the configuration of a multi-participation remote collaboration system according to an embodiment of the present invention;
3 is a diagram showing the configuration of an operator terminal according to an embodiment of the present invention;
4 is a diagram showing the configuration of a remote server according to an embodiment of the present invention;
5 is a view showing an augmented reality sharing environment for providing technical support of experts according to an embodiment of the present invention;
6 is a diagram illustrating information transmitted and received between a worker terminal screen and a subject in order to show an example of augmented reality sharing-based multi-participation remote collaboration according to an embodiment of the present invention;
7 is a diagram illustrating an example of AR sharing-based multi-participation remote collaboration based on a usage location for each subject according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments make the disclosure of the present invention complete, and the general knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to the possessor, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same elements throughout the specification.

In describing the embodiments of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted, and the terms to be described later are used in the embodiment of the present invention. These terms are defined in consideration of the function of the user, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout the present specification.

Each block in the accompanying block diagram and combinations of steps in the flowchart may be executed by computer program instructions (execution engine), which are executed by the processor of a general-purpose computer, special-purpose computer, or other programmable data processing device. It may be mounted so that its instructions, which are executed by the processor of a computer or other programmable data processing device, create means for performing the functions described in each block of the block diagram or in each step of the flowchart.

These computer program instructions may also be stored in a computer-usable or computer-readable memory that may direct a computer or other programmable data processing device to implement a function in a particular manner, and thus the computer-usable or computer-readable memory. It is also possible for the instructions stored in the block diagram to produce an article of manufacture containing instruction means for performing the functions described in each block of the block diagram or each step of the flowchart.

And since the computer program instructions may be mounted on a computer or other programmable data processing device, a series of operational steps is performed on the computer or other programmable data processing device to create a computer-executed process to create a computer or other programmable data processing device. It is also possible that the instructions for performing the data processing apparatus provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

In addition, each block or step may represent a module, segment, or portion of code comprising one or more executable instructions for executing specified logical functions, and in some alternative embodiments the blocks or steps referred to in the block or steps. It should be noted that it is also possible for functions to occur out of sequence. For example, it is possible that two blocks or steps shown one after another may be performed substantially simultaneously, and also the blocks or steps may be performed in the reverse order of the corresponding functions, if necessary.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention illustrated below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art to which the present invention pertains.

1 is a diagram illustrating an operator terminal screen in an augmented reality-based remote guidance environment according to an embodiment of the present invention.

1 , Augmented Reality (AR, hereinafter referred to as 'AR')-based remote guidance (Remote Guidance) combines a virtual AR manual with an image captured by a worker at a work site in real time. By providing a guide screen, it means a service that guides workers so that they can work on their own according to the AR manual. AR manual is a virtual AR content that provides work step information for the operator to operate the device. For example, it can be expressed in various ways such as a text-type work order, an image of the same shape as the work target, arrows, numbers, colors, images, and the like.

AR guide screen For example, when it is assumed that a problem has occurred in the power supply of the power supply in the power panel at the work site, the screen of the operator terminal 2 displays "The power of the power supply of the power panel exceeds the threshold value, so The status information 102 is displayed, such as a warning message stating that it should be reduced. Then, identification information 100 such as an identifier such as a block type is displayed on the screen for the power panel cover in which the problem has occurred so that the operator recognizes that the power panel cover is a work target. The object of the identification information 100 may be changed according to the AR manual.

In addition, the AR manual 104 is also displayed on the screen. For example, "1. Open the power panel cover. 2. Lower the breaker. 3. Replace the power supply. 4. Raise the breaker." A preset AR manual is displayed on the screen according to the work order. In "1. Open the power panel cover.", the handle of the power panel cover in the captured image may be identifiable by a dotted line or the like. In "2. Lower the breaker.", the breaker in the captured image may be identifiable by a dotted line or the like. In "3. Replace the power supply.", the power supply in the captured image may be identifiable by a dotted line or the like. Although the AR manual 104 in FIG. 1 is shown in text form, it is content that can be provided in various formats such as images and videos as well as text.

With AR-based remote guidance, the operator can easily and conveniently perform the operation without the help of an experienced expert on the device or even without the operation manual. The device includes equipment, machines, facilities, etc. installed in the work site, and may include parts, modules, etc. constituting one device. Operations include control, remote repair, maintenance, system maintenance, adjustment, and operation. For example, the task may be repairing a problematic device or taking action before a problem occurs on a potentially problematic device.

A work site may be an industrial site such as a factory (smart factory) or a building (smart building), or it may be a home (smart home). For example, in an industrial field, AR-based remote guidance can be provided for air conditioners, ventilation, air conditioners, fans, boilers, cooling towers, pumps, temperature/humidity sensors, refrigerators, lighting equipment, power equipment, and fire systems in factories. Mechanical equipment, lighting, power, access control, disaster prevention, parking management, and facility management are possible while comprehensively controlling and monitoring the situation of the entire building through the remote server (3). Operators can use AR-based remote guidance information to perform tasks smoothly without the assistance of experts.

AR-based remote guidance can be operated in a home environment. For example, through the remote server 1, home appliances such as TVs, air conditioners, and refrigerators in the home, energy consumption devices such as water, electricity, and air conditioning, security devices such as door locks and surveillance cameras, expensive cars and electric vehicles, etc. It can be remotely adjusted while monitoring and controlling the devices of In addition, when a problem occurs in the corresponding home device, if the consumer has only the worker terminal 2, AR-based remote guidance for solving the problem can be provided, so that the problem of the device can be easily solved. In this case, the consumer does not need to contact the customer center to call an expert or to bear travel expenses separately, and time can be prevented. In addition, there is no need for an immersive display that requires an existing separate hardware such as smart glass, and it can be combined and used.

AR-based remote guidance can be provided for multiple IoT devices used in smart devices, smart buildings, and smart factories. The IoT device may be a digital signage, a smart TV, a smart watch, a smart CE, a drone, or the like. The multi-participation remote collaboration system 1 according to another embodiment may provide AR-based remote guidance for legacy devices manufactured before the IoT era without IoT sensing capability. For example, AR-based remote guidance using the operator terminal 2 can be provided even for legacy devices that cannot sense information about the device and its surrounding environment because there is no IoT sensor built-in or mounted in the device.

2 is a diagram illustrating the configuration of a multi-participation remote collaboration system according to an embodiment of the present invention.

Referring to FIG. 2 , the multi-participation remote collaboration system 1 includes a worker terminal 2 , a remote server 3 and an expert terminal 4 .

The operator terminal 2 is a user terminal possessed by a worker who works at a work site. The operator terminal 2 acquires a photographed image of the work site, and combines the acquired image with a virtual AR manual and displays it on the screen. The worker terminal 2 may recognize the contour of the work target through AR-based object recognition and receive the machine learning-based object recognition result received from the remote server 3 to further recognize parts and modules. When the operator terminal 2 is connected to the remote server 3 , the operator terminal 2 becomes an agent that performs the control command received from the remote server 3 . The worker terminal 2 may run an App for remote guidance. In this case, it has a form that can support all different OSs, such as Android, iOS, and Windows, which are different from each other.

The remote server 3 monitors devices located at the work site and performs AR-based remote guidance for devices that require work. Through the AR manual provided through the operator terminal 2, the operator may not be able to solve the problem, so a situation may arise that requires the help of an expert. In this case, the operator terminal 2 may be connected to the expert terminal 4 through the remote server 3 to receive technical support from the expert. To this end, the remote server 3 provides a common AR manual to the operator terminal 2 and the expert terminal 4, and when receiving an expert command for technical support from the expert terminal 4, the operator terminal according to the received command (2) is controlled remotely.

The expert terminal 4 is a terminal possessed by an expert for remotely providing technical support to a worker, and multiple participation is possible. The experts include not only experts in each field, but also technical engineers of manufacturing facilities. Smart remote support for the operator terminal 2 is possible by collaborating between each expert terminal 4-1, 4-2, ..., 4-n. For example, a given expert terminal shares an AR manual with the operator terminal 2 and other expert terminals connected to the call, respectively, and provides technical support for the operator's work based on the AR manual.

The multi-participation remote collaboration system 1 according to an embodiment enables technical support by maximizing the AR function through multi-party participation of experts, out of the existing simple remote support in a remote guidance environment. For example, many experts can share AR manuals based on AR and provide technical support to workers using the shared AR manuals, thereby maximizing the effect of using AR.

When remote support is required for work on the device at the work site, the worker proceeds with the work according to the AR manual provided to his/her operator terminal 2 . However, there may be situations in which the worker needs the help of a professional during the operation. In this case, it is necessary to seek the support of the primary technical channel of experts, eg technical engineers.

In general, when an operator seeks expert help, the operator must remotely listen to the expert for re-explanation to solve the task problem, separate from the AR guide. In this case, work efficiency is reduced. In particular, in the case of a technical trainee or an on-site user training environment, the worker is more ineffective.

The multi-participation remote collaboration system 1 according to an embodiment can easily handle the above-described problem by sharing AR manuals between workers and experts and between experts.

AR-based remote guidance between workers and experts For example, the technical engineer at the first technical counter listens to the worker’s situation, identifies the wrong part, returns to a specific location in the AR manual that consists of multiple work steps, and then returns to the wrong location. You can proceed with AR-based remote guidance from the beginning. In this case, detailed remote guidance can be provided by providing Annotation Tool and Direction Tool together.

Depending on the situation, experts can communicate through a mutual call connection and share AR manuals. In addition, since multiple experts can participate at the same time, remote guidance can be provided through remote collaboration. In-depth technical support is also possible through remote guidance, where experts provide audio and AR images to workers.

When necessary, if you need to follow the guide of an external/in-house expert of a site (branch, etc.) that is operating normally, you can share the AR guide screen including the AR manual from the worker terminal (2) to the expert terminal (4) in the reverse direction. . In this case, the operator terminal 2 may be remotely controlled according to the expert command of the expert terminal 4 .

The above-described components may communicate through a network, and the network may be a wired network such as a local area network (LAN), a wide area network (WAN) or a value added network (VAN), or the like; It may be implemented as any type of wireless network such as a mobile radio communication network or a satellite communication network.

3 is a diagram illustrating a configuration of an operator terminal according to an embodiment of the present invention.

2 and 3 , the operator terminal 2 includes a data acquisition unit 20 , an input unit 21 , a control unit 22 , an output unit 23 , a communication unit 24 , and a storage unit 25 . do.

The data acquisition unit 20 acquires a photographed image taken in real time based on the current point in time at which the worker looks at the work object at the work site. The data acquisition unit 20 may acquire a captured image from a camera. The data acquisition unit 20 transmits the acquired captured image to the control unit 22 . The data acquisition unit 20 may use remote guidance through an App installed in the worker terminal 2, and in this case, may acquire a real-time captured image through the app camera. While the operator is using the AR-based remote guidance, the footage is captured in real time.

The input unit 21 receives a user manipulation signal. The user manipulation signal includes a user's screen touch signal, a key input signal, a voice signal, and a biosignal. The input unit 21 transmits the received user manipulation signal to the control unit 22 .

The output unit 23 outputs the information processed through the control unit 22 on the AR guide screen. The output information includes a real object in the captured image and a virtual object in the AR manual. The AR manual can guide the operator's work by combining a virtual object of the same shape as the real object to be worked with the captured image. At this time, the operator can easily perform the task by aligning the location of the real object in the captured image with the virtual object in the AR manual and manipulating the state of the real object in the captured image so that the state becomes the same as the virtual object in the AR manual. have. As an example of the output, an image may be displayed on a screen, and an audio may be output through a speaker.

The virtual object displayed through the output unit 23 also has identifier information for identifying a work target. For example, an identifier such as an arrow, a line, or a block is displayed on the screen. In this case, the work target may be changed according to the AR manual. Since the real screen shot in real time continues to move, the identifier also moves according to the moving object. The output unit 23 may inform the position of the work target. For example, the shape of a hand indicates a part that needs work.

The output unit 23 may display various status information about the device, such as a control state of a work target, a failure state, an operation state, time information, and a control parameter required to work in the device. At this time, detailed description including the part number of the work target, life cycle, recent replacement history, breakdown history, etc. may be displayed on the screen. For example, when the device is an air conditioner, the control parameter may be a target temperature (or set temperature), an upper limit temperature, a lower limit temperature, an air volume, whether to operate or stop. If the device is a lighting fixture, the control parameter may be lighting brightness or the like. The output unit 23 may automatically display status information for a device requiring work, but may also display status information at the request of an operator. For example, when the operator selects a predetermined device on the screen, various status information about the selected device may be displayed on the screen.

The output unit 23 may display a virtual object for informing a sequential AR manual on the screen. At this time, it is possible to inform the operator through the output unit 23 of the device or parts in the device that require work in accordance with the AR manual.

The output unit 23 may display a virtual object for informing the user of a tool required for the operation on the screen. The working tools include, for example, a hammer, a screwdriver, a hand, and the like. The output unit 23 may display a process marker on the screen that allows the user to grasp the state of the step for the AR manual composed of a plurality of work steps. The process marker includes identifier information for each AR manual, identifier information for each work step in the AR manual, and marker information assigned to at least one work step identifier based on the current work step for access to the work step desired by the user. do.

The communication unit 24 is network-connected to the remote server 3 to perform wired/wireless communication. The worker terminal 2 may be connected to the remote server 3 through a dedicated network through the communication unit 24 or may be connected through a general-purpose network such as the Internet.

The communication unit 24 according to an embodiment requests and receives the operation manual from the remote server 3 . The communication unit 24 may start the operation guidance upon receiving the operation command from the remote server 3 . For example, when the remote server 3 monitors devices in the work site and an error occurs or a device with a possibility of an error is detected, a work request command for the device is transmitted to the worker terminal 2 and the operator inform

The communication unit 24 according to an embodiment transmits and shares the captured image acquired through the data acquisition unit 20 to the remote server 3 . And the object recognition result can be exchanged with the remote server 3 . For example, an object recognition result through AR-based image analysis of the worker terminal 2 may be transmitted to the remote server 3 , and an AI-based object recognition result may be received from the remote server 3 .

The storage unit 25 stores various data. The data may be unique information, location information, drawings of the devices, work manual information, parts information in the device, work schedule information, and the like of devices on the job site. Such information may be downloaded from the remote server 3 and stored, or may be input from a user through the input unit 21 .

The storage unit 25 is a hard disk (Hard Disk), as well as built-in memory such as flash memory (Flash Memory), CF card (Compact Flash Card), SD card (Secure Digital Card), SM card (Smart Media Card), MMC card (Multi-Media Card) or a memory stick (Memory Stick) may be implemented as a memory when moving, such as inside the operator terminal 2, it may be provided in a separate device.

The control unit 22 controls the operation of each configuration described above. The control unit 22 according to an embodiment repeatedly performs the sequential process of object recognition → object tracking → object rendering on the captured image obtained through the data obtaining unit 20 to link the AR manual to the captured image. .

The control unit 22 according to an embodiment includes an image recognition unit 220 , a tracking unit 222 , and a rendering unit 224 .

The image recognition unit 220 recognizes a work target based on AR and artificial intelligence (AI, hereinafter referred to as 'AI') in the captured image. The tracking unit 222 tracks the work object recognized through the image recognition unit 220 . Since the photographed image captured through the camera is not fixed but continues to move due to the movement of the operator, the tracking unit 222 performs an operation of continuously following the movement of the object recognized in the photographed image. The rendering unit 224 calculates the interaction between the real object and the virtual object and reflects the calculation result to match the virtual object to the captured image to help the work on the work target and then render, or render the virtual object and then shoot The AR manual is created by matching the image. In this case, the location of the virtual object may be selected and placed in the captured image.

The image recognition unit 220 according to an embodiment uses AR-based image analysis and AI-based machine learning for object recognition. Image analysis may use a general algorithm for recognizing an object in the acquired image. For example, an object is recognized using point cloud information, depth information, color information, and the like in an image. Since image analysis-based object recognition is a known technology, a detailed description thereof will be omitted. In this case, the object recognition rate may be increased by image processing, for example, image binarization, edge blending, masking, and the like.

Machine learning is one of the methods used in AI, and it is a technology that learns by inputting a lot of data into a computer and classifying similar ones. For example, if a picture similar to a stored picture of a dog is input, the computer classifies it as a picture of a dog. Objects in the image can be recognized using the learning data accumulated through machine learning. Machine learning may use SVM (Support Vector Machines), Deep Learning, and the like.

The image recognition unit 220 intends to increase the accuracy of object recognition by utilizing such AI-based object recognition as an auxiliary means of AR-based image analysis. For example, the image recognition unit 220 is an AR-based object recognition result through the worker terminal 2, an AI-based object recognition result through the worker terminal 2, or the remote server 3 received through the communication unit 24 ) to increase the object recognition accuracy by reflecting the AI-based object recognition results.

The remote control processing unit 226 generates a remote control request message and transmits it to the remote server 3 through the communication unit 23 to request remote control. In this case, the AR guide screen may be shared remotely under the control of the remote server 3 . The remote control processing unit 226 is an expert terminal through the remote server (3) when a situation occurs in which the worker needs help from an expert while the worker is working according to the AR manual provided to his/her operator terminal (2) You can request technical support in (4).

The device control unit 228 processes a remote control command of the remote server 3 . The remote control command may be an expert command for technical support received from the expert terminal 4 .

4 is a diagram illustrating a configuration of a remote server according to an embodiment of the present invention.

2 and 4 , the remote server 3 includes an object recognition unit 300 , a manual providing unit 302 , a status tracking unit 304 , a technical support unit 306 , a collaboration support unit 308 , and an activity management unit. 310 , a network connection unit 312 , a monitoring unit 314 , a client management unit 316 , a data analysis unit 318 , and a protocol processing unit 320 . The remote server 3 may be connected by establishing a database storing information necessary for remote control.

The object recognition unit 300 recognizes an object through machine learning based on the learning data for the captured image received from the operator terminal 2 through the network connection unit 312 . The machine learning-based object recognition result is transmitted to the operator terminal 2 through the network connection unit 312 .

The state tracking unit 304 tracks the use positions of the operator terminal 2 and the expert terminal 4 . The use position of the operator terminal 2 is the operator's current work stage in the AR manual. The use position of the expert terminal 4 is an expert command input from the expert.

The network connection unit 312 maintains homeostasis of the network connection regardless of the type of network. The network connection unit 312 according to an embodiment transmits and receives information to and from the operator terminal 2 of the operator and the expert terminal 4 of the expert. For example, the network connection unit 312 receives and shares a captured image from the worker terminal 2 , receives object information recognized through image analysis based on AR of the worker terminal 2 , and machine learning-based object recognition The result is transmitted to the operator terminal 2 . In addition, the AR manual is shared with the expert terminal 4 . When the network connection unit 312 receives a remote control request from the operator terminal 2 , it shares the screen of the operator terminal 2 , and transmits a remote control command to the operator terminal 2 through the network connection unit 312 .

The network connection unit 312 according to an embodiment receives the operator's use location information from the operator terminal 2 and transmits it to the expert terminal 4 , and receives the expert's use location information from the expert terminal 4 . In addition, it receives an expert command for technical support for the operator from the expert terminal 4 and transmits it to the operator terminal 2 .

The network connection unit 312 according to an embodiment shares the AR guide screen including the AR manual of the worker terminal 2 in the reverse direction to the expert terminal 4 of the site that is operating normally. The site may be a branch office, etc., the expert is an in-house or external expert, and the worker performs the work according to the expert's guide.

The monitoring unit 314 monitors the screen of the operator terminal 2 of the operator at the work site to determine the operation state of the AR manual. At this time, it is possible to check whether the AR manual operates normally.

The client management unit 316 receives the real-time status information of the client and checks the status of the client. If the client is working normally or not, figure out what kind of problem there is and deal with it. The client may be a worker terminal 2 , an app running on the worker terminal 2 , or various devices in the worksite. The client management unit 316 according to an embodiment monitors the devices in the job site to predict an error in advance and generates a job command for handling the error for the device before the error occurs. The generated work command is transmitted to the operator terminal 2 of the operator through the network connection unit 312 .

The data analysis unit 318 analyzes the use information for the operator terminal 2 of the operator to determine the operator state. For example, it is possible to know whether the worker has mastered the app usage frequency and the worker's work from information using the worker terminal 2 .

Technical support unit 306 at the request of the operator or by determining whether the operator additionally requires technical support by an expert, and when necessary, requests the expert terminal 4 to provide the technology to the operator terminal 2 remotely from the expert Support. The technology supported by the technical support unit 306 is a work step approach technology that approaches a predetermined work step in the AR manual, a synchronization technology that synchronizes the AR manual between the worker terminal 2 and the expert terminal 4, and the AR manual is initialized There are an initialization technology that is used, a tool use technology that supports technology through a tool including an annotation tool or a direction tool, and a technology that supports technology through video, chatting, or voice. At this time, the predetermined work step approach is based on the current work step, moving the work step in the reverse direction, moving the work step in the forward direction, repeating the current work step, moving to the first work step, or moving to another augmented reality manual. may include

The technical support unit 306 according to an embodiment includes the AR manual operation state identified through the monitoring unit 314 , the operator terminal state identified through the client management unit 316 , and the operator state identified through the data analysis unit 318 . Use at least one of the following to determine whether an expert provides technical support and select an expert to request technical support.

The collaboration support unit 308 relays technical support collaboration between a plurality of experts through a plurality of expert terminals 4-1, 4-2, ..., 4-n. For example, the collaboration support unit 308 provides a teleconference function in which a plurality of experts can participate simultaneously. At this time, the collaboration support unit 308 may receive the voice along with the augmented reality image from each expert terminal (4-1, 4-2, ..., 4-n) during a teleconference and deliver it to the worker terminal (2). .

The collaboration support unit 308 defines the expert command priority in advance, and when a collaboration between the expert terminals 4-1, 4-2, ..., 4-n is performed, a plurality of expert terminals 4-1, 4-2, …, 4-n) together with expert commands, the expert commands can be sequentially processed according to a predefined command priority. As another example, when the collaboration support unit 308 receives an expert command from a plurality of expert terminals 4-1, 4-2, ..., 4-n, the corresponding expert terminals 4-1, 4-2, ... , 4-n) can mediate negotiations among themselves.

The activity manager 310 may record an activity in a preset time unit and cancel and reverse the activity according to an expert command of the expert terminal 4 .

The protocol processing unit 320 is a software module that receives and parses the client's normal signal and implements a communication protocol for processing it.

5 is a diagram illustrating an AR sharing environment for providing technical support from an expert according to an embodiment of the present invention.

Referring to FIGS. 2 and 5 , while performing AR-based remote guidance, a procedure for operating a device on the job site is provided to the operator terminal 2 as an AR manual. In the middle of the work, if the worker's understanding of the work is low or the work is complicated, technical support from experts can be processed in real time. When the task is complicated, it is a product that requires multiple manipulations, and it is necessary to repeat the steps of the AR manual with a very high number of processing and manipulations due to the high level of difficulty in each step, or to try re-manipulating the AR manual in reverse.

The request for technical support may be directly requested by the operator, and the remote server 3 may determine whether the expert provides technical support and, if necessary, request the expert. For the determination of the remote server 3, at least one of the AR manual operation state, the state of the operator terminal 2, and the operator state may be used.

In the AR-based remote guidance environment, it is possible to provide technical support by maximizing the AR function through the multilateral participation of experts, breaking away from the existing simple remote support. For example, multiple experts can share AR manuals and technically support workers using the shared AR manuals. At this time, AR manuals can be shared between workers and experts, and between experts.

As shown in FIG. 5, AR sharing between workers and experts, for example, expert #1 (eg, a technical engineer at the primary technical window) (#5-1) listens to the operator's situation, and then the wrong part AR-based remote guidance can be carried out, starting with the wrong part, after identifying and returning to a specific location in the AR manual, which consists of multiple work steps. In this case, detailed AR-based remote guidance can be performed by providing an annotation tool and a direction tool together (Annotation Tool & Direction Tool).

Depending on the situation, experts can communicate through a mutual call connection and share AR manuals. For example, as shown in FIG. 5, expert #1(5-1) requests sharing to expert #2(5-2), and again expert #2(5-2) sends expert #2(5-2) to expert #2(5-2). 3(5-3) can request to share.

Each expert (5-1, 5-2, ..., 5-n) can remotely control the operator while sharing the AR manual (AR sharing & Control). In addition, since multiple experts (5-1, 5-2, ..., 5-n) can participate simultaneously, AR-based remote guidance can be provided through remote collaboration. In-depth technical support is also possible through AR-based remote guidance, in which experts provide voices along with AR images to workers.

When necessary, if you need to follow the guide of an external/in-house expert of a site (branch, etc.) that is operating normally, you can share the AR guide screen including the AR manual from the worker terminal (2) to the expert terminal (4) in the reverse direction. . In this case, the operator terminal 2 may be remotely controlled according to the expert command of the expert terminal 4 (Follow & Check).

6 is a diagram illustrating information transmitted/received between a screen of a worker terminal and a subject in order to show an example of AR sharing-based multi-participation remote collaboration according to an embodiment of the present invention.

2 and 6 , the operator terminal 2 acquires a photographed image 600 of a work site, recognizes an object 610 as a work target from the obtained photographed image 600, and recognizes the recognized object 610 ) to generate a virtual AR manual 620 and display it on the AR guide screen. In this case, in addition to the AR manual 620 , virtual work guidance information including identifier information, location information, state information, work tool information, and the like for a work target may be additionally displayed on the screen. The identifier information is a virtual object for identifying a work target. For example, the work target is indicated by indicators such as dotted lines or blocks. The location information is a virtual object for informing the location of the work target. For example, a virtual finger indicates the location of a device that requires work or a part within the device. The AR manual 620 is a virtual object that guides sequential operation steps. The work tool information is a virtual object for informing the tool required for work. It displays all the virtual tools needed for the job, for example a screwdriver, hammer, etc.

Referring to FIG. 6 , if an expert's support is additionally required while the worker is working according to the AR manual 620 while looking at the AR guide screen of the worker terminal 2, the remote server 3 performs AR according to the worker's request. The manual 620 may be transmitted to the expert terminal #1(4-1) possessed by the expert #1(5-1). With the expert #2(5-2), the AR guide screen of the worker terminal 2 may be transmitted to the expert terminal #2(4-2).

Specialists can provide a variety of technical assistance to workers in need. For example, as shown in FIG. 6 , during a teleconference, expert #3(5-3) may transmit a voice to the operator terminal 2 through expert terminal #3(4-3). As another example, expert #n(5-n) may support an annotation tool or a direction tool through the expert terminal #n(4-n). The annotation tool includes tools such as drawing, line drawing, point, and text, and a drawing tool 630 is shown in FIG. 6 .

7 is a diagram illustrating an example of AR sharing-based multi-participation remote collaboration based on a usage location for each subject according to an embodiment of the present invention.

2 and 7, the remote server 3 is a status tracing module (Status Tracing Module) (304 in FIG. 4) through the operator terminal 2 and each expert terminal (4-1, 4-2, 4) -3) Track the location of the star. The use position of the operator terminal 2 is the current work stage (eg, U#1001_G#101_#103) of the operator (User #1001) in the AR manual. Here, G#101 is the serial number of the AR manual, and #103 is the sequence number of the work step. The use location for each expert terminal is an expert command input from the expert. For example, the location of expert #1 (specialist #2001) is U#1001_G#101_Sync, the location of expert #2 (specialist #2002) is U1001_G#101_discussion, and the location of expert #3 (specialist #2003) The usage location is U#1001_G#101_Request reset. Sync means a synchronization command, discussion means a voice conference command, and Request reset means an initialization request command.

The remote server 3 matches and stores the subject and use location information. For example, match and save U1001 - #103, S#2001 - #Sync, S#2002 - #discussion, S#2003 - #Request reset. The remote server 3 delivers the use location information (U1001 - #103) of the worker (U#1001) to each expert (S#2001, S#2002, S#2003), and each expert (S#2001) , S#2002, S#2003) can receive expert commands (#Sync, #discussion, #Request reset) to remotely control the operator terminal 2 according to the expert command. The expert command may be, in addition to synchronization (#Sync), voice conference (#discussion), and reset request (#Request reset), work step access, tool use, video support, chat support, and the like.

When a plurality of experts perform remote collaboration through multiparticipation, if there are a plurality of expert commands to be processed, the remote server 3 may sequentially process expert commands according to a preset command priority. Alternatively, it can be brokered through consultation with experts. The remote server 3 records the activity by time period, and may cancel or reverse the activity by an expert. The operator can capture the AR guide screen and deliver the captured AR guide screen and related information to the expert.

So far, the present invention has been looked at around the embodiments. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the above description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (11)

A worker terminal that acquires an image captured in real time of a worker's on-site situation, recognizes a work object from the obtained captured image, and combines the augmented reality manual with the captured image including the recognized work object;
at least one expert terminal that shares an augmented reality manual with a worker terminal and other expert terminals, respectively, and provides technical support for a worker's work based on the augmented reality manual; and
A remote server that provides a common augmented reality manual to the worker terminal and each expert terminal and remotely controls the worker terminal according to the received command when receiving an expert command for technical support from the expert terminal;
Augmented reality sharing-based multi-participation remote collaboration system comprising a. The method of claim 1, wherein the remote server is
a state tracking unit that tracks the use location of each operator terminal and each expert terminal;
a network connection unit for receiving the use location information of the operator from the operator terminal, transmitting it to each expert terminal, and receiving the use location of the corresponding expert from each expert terminal; and
a technical support unit that remotely supports the operator's terminal from the operator at the request of the operator or as the operator receives an expert command by requesting the expert terminal when necessary by determining whether technical support by an expert is additionally required;
Augmented reality sharing-based multi-participation remote collaboration system comprising a. The method of claim 2,
The use position of the worker terminal is the current working stage of the worker in the augmented reality manual, and the use position for each expert terminal is an expert command input from the expert. Augmented reality sharing-based multi-participation remote collaboration system, characterized in that. 3. The method of claim 2, wherein the remote server is
A monitoring unit for monitoring the operator terminal screen of the operator to determine the operation state of the augmented reality manual;
A client management unit for receiving real-time status information on the worker terminal to determine the worker terminal status; and
A data analysis unit that analyzes the operator's terminal usage information of the operator to determine the operator status; It further includes,
technical support department
Using at least one of the augmented reality manual operation state identified through the monitoring unit, the worker terminal state identified through the client management unit, and the worker state identified through the data analysis unit, determine whether the expert will provide technical support and select an expert to request technical support Augmented reality sharing-based multi-participation remote collaboration system, characterized in that 3. The method of claim 2, wherein the technical support
A work step approach technology that approaches a predetermined work step in the augmented reality manual, a synchronization technology that synchronizes the augmented reality manual between the worker terminal and the expert terminal, an initialization technology that initializes the augmented reality manual, an annotation tool or a map tool ( Direction tool) using tools that support technology through tools including augmented reality sharing-based multi-participation remote collaboration system, characterized in that it supports at least one technology of technology supporting technology through video, chatting, or voice. The method of claim 5,
Predetermined work step approach includes moving a work step in the reverse direction based on the current work step, moving a work step in the forward direction, repeating the current work step, moving to the first work step, or moving to another augmented reality manual. Augmented reality sharing-based multi-participation remote collaboration system, characterized in that. The method of claim 1, wherein the remote server is
a collaboration support unit that relays technical support collaboration between a plurality of experts through a plurality of expert terminals;
Augmented reality sharing-based multi-party remote collaboration system, characterized in that it comprises a. The method of claim 7, wherein the collaboration support unit
Provides a remote conference function that allows multiple experts to participate simultaneously,
Augmented reality sharing-based multi-participation remote collaboration system, characterized in that the voice is received along with the augmented reality image from each expert terminal during a teleconference and delivered to the worker terminal. The method of claim 7, wherein the collaboration support unit
Augmented reality sharing, characterized in that the expert command priority is defined in advance, and expert commands are sequentially processed according to the predefined command priority when receiving expert commands from a plurality of expert terminals during collaboration between expert terminals based multi-participant remote collaboration system. The method of claim 7, wherein the collaboration support unit
Augmented reality sharing-based multi-participation remote collaboration system, characterized in that when expert commands are received together from a plurality of expert terminals, the corresponding expert terminals are negotiated. The method of claim 1, wherein the remote server is
Augmented reality sharing base, characterized in that after sharing the augmented reality guide screen including the augmented reality manual of the worker terminal in the reverse direction to the expert terminal of the site that is operating normally, the operator terminal is remotely controlled according to the expert command of the expert terminal Multi-Participant Remote Collaboration System.

Images