KR102132330B1 - Remote guidance apparatus and method capable of handling hyper-motion step based on augmented reality and machine learning - Google Patents

Abstract

Disclosed is an augmented reality and machine learning based remote guidance device capable of processing hypermotion steps and a method therefor. The remote guidance device provided in the user terminal according to an embodiment of the present invention includes an input unit that receives a current image photographing a field situation of an operator, and recognizes a work object from the current image received from the input unit, and recognizes the current work object. And a control unit that provides a corresponding augmented reality manual by matching a virtual work manual to a work step corresponding to a user manipulation among a plurality of work steps constituting the augmented reality manual according to a user manipulation signal. , An output unit displaying an augmented reality manual screen.

Description

Remote guidance apparatus and method capable of handling hyper-motion step based on augmented reality and machine learning}

The present invention relates to augmented reality based image analysis and remote control technology.

In order to maintain the equipment, many years of experience are required, and even a skilled person takes a long time to understand it when introducing new equipment. Also, no matter how skilled, it is difficult to know all the maintenance procedures, and it is difficult to predict when and what maintenance needs will occur. Accordingly, mechanics have carried out maintenance work by always carrying a manual for equipment maintenance.

Manuals for maintenance are generally produced in books or 2D images. The problem with manuals made of books is that even if one part fails, the corresponding page must be found individually, and if the manual is old, the color is changed or the damage is severe. In the case of a complex type of mechanical equipment such as an aircraft, hundreds of thousands of manuals are required, so the amount is vast and cannot be carried.

Problems such as maintenance are not limited to industrial sites. For example, in the case of a refrigerator failure in a home, the consumer is often unable to fix it himself, and often requires a separate specialist to repair it. At this time, it takes a lot of time and money to solve the problem, which is inefficient.

According to one embodiment, even if it is a complex structured equipment or an unskilled worker, it is possible to work quickly and easily according to the sequential instructions provided based on augmented reality, and it can be operated quickly and easily even when multi-step operation is required. We propose an augmented reality and machine learning-based remote guidance device capable of processing motion steps and a method therefor.

The remote guidance device provided in the user terminal according to an embodiment of the present invention includes an input unit that receives a current image photographing a field situation of an operator, and recognizes a work object from the current image received from the input unit, and recognizes the current work object. And a control unit that provides a corresponding augmented reality manual by matching a virtual work manual to a work step corresponding to a user manipulation among a plurality of work steps constituting the augmented reality manual according to a user manipulation signal. , An output unit displaying an augmented reality manual screen.

The user manipulation signal may include at least one of a user's screen touch signal, audio signal, image signal, and bio signal. The access to the work step corresponding to the user operation can be performed by moving the work step in the reverse direction based on the current work step, moving the work step in the forward direction, repeating the current work step, moving to the first work step or other augmented reality manual. It may include the movement of.

The output unit displays a process marker on the screen for augmented reality manual composed of a plurality of work steps to determine the step status, and the process marker includes identifier information for each augmented reality manual and identifier information for each work step in the augmented reality manual. , May include marker information assigned to at least one work step identifier based on the current work step for access to a work step desired by the user.

The input unit includes a touch input unit that receives a screen touch signal from a user, and the control unit receives a user touch signal from the touch input unit targeting the process marker displayed on the screen, recognizes the user touch, and configures augmented reality manual. It may include a touch recognition unit that accesses to a work step corresponding to the recognized touch and provides a corresponding work step screen.

The input unit includes a voice input unit that receives a voice signal from a user, and the control unit allocates a voice recognition code having identifier information and location information to work steps constituting the augmented reality manual, and receives a user voice signal from the voice input unit. It may include a voice recognition unit for recognizing a user's voice and accessing a work step to which a voice recognition code corresponding to the recognized voice is assigned among a plurality of work steps to provide a corresponding work step screen.

The input unit includes a bio-input unit that receives a bio-image photographed for a user; Including, the control unit assigns a biometric code having identifier information and location information to work steps constituting the augmented reality manual, receives the user's biometric image from the biometric input unit, detects the user's face and body, and detects the detected face And a biometric recognition unit that recognizes an expression or a motion from the body information and accesses a work step to which a biometric code corresponding to the recognized expression or motion is assigned among a plurality of work steps to provide a corresponding work step screen.

The input unit includes an image input unit for acquiring a current situation image of a work site in real time, and the control unit recognizes an object in the current image based on augmented reality, tracks the movement of the recognized object, and tracks the actual object to be tracked. It may include an image recognition unit that combines and renders virtual objects corresponding to augmented reality-based work guidance information, or renders virtual objects and combines them with real objects to generate an augmented reality manual. The remote guidance device is networked with the remote server and transmits and shares the current image acquired through the input unit to the remote server, transmits the object information recognized through the augmented reality based image analysis of the remote guidance device to the remote server, and sends the machine from the remote server. Further comprising a communication unit for receiving the object information recognized through the learning, the image recognition unit reflects the machine learning-based object recognition result of the remote server received from the communication unit in the contour recognition result of the augmented reality-based object, the contour of the augmented reality-based object Objects not recognized at the time of recognition can be recognized.

The remote guidance device further includes a process marker provided in the work object and a communication unit capable of short-range wireless communication, and the control unit recognizes the process marker by the radio signal received from the communication unit and identifies the work step recorded in the recognized process marker, It may include a wireless signal recognition unit to access the desired work step by the user manipulation during the identified work step to provide a screen of the work step.

In a remote guidance method using a remote guidance device provided in a user terminal according to another embodiment, the remote guidance device: acquires a current image of a worker's field situation, recognizes a work object from the current image, and recognizes the recognized work object. Tracking, acquiring a work manual for guiding a worker's work on the tracked work target, and matching the received virtual work manual to the current image including the tracked work target to provide an augmented reality manual And receiving a user operation signal to recognize a user operation and providing a corresponding operation step screen among a plurality of operation steps constituting the augmented reality manual.

The method for providing remote guidance further includes displaying a process marker on the screen for augmented reality manual composed of a plurality of work steps to identify a step status, and the process marker includes identifier information for each augmented reality manual and augmented reality It may include identifier information for each work step in the manual, and marker information assigned to at least one work step identifier based on a current work step for access to a work step desired by a user.

In the step of providing the corresponding work step screen, the process marker displayed on the screen receives a touch signal from the user, recognizes the user touch, and is a work step corresponding to the recognized touch among a plurality of work steps constituting the augmented reality manual. You can access and provide the corresponding work step screen.

The method for providing remote guidance further includes assigning a voice recognition code having identifier information and location information to work steps constituting the augmented reality manual, and inputting a voice signal from a user in the step of providing a corresponding work step screen. Receiving a user's voice and recognizing a voice recognition code corresponding to the recognized voice among a plurality of work steps may be approached to the assigned work step to provide a corresponding work step screen.

The method for providing remote guidance further includes assigning a biometric code having identifier information and location information to work steps constituting the augmented reality manual, and in the step of providing a corresponding work step screen, photographing the user Receives the received biometric image, detects the user's face and body, recognizes facial expressions and motions from the detected face and body information, and accesses to a work step in which a biometric code corresponding to the recognized facial expression or motion is assigned among a plurality of work steps Thus, a corresponding work step screen can be provided.

The method for providing remote guidance includes the step of short-range wireless communication with a process marker provided on a work target, and recognizes a process marker by a radio signal received through wireless communication, grasps a work step recorded in the recognized process marker, and is identified. During the work step, the user may further include a step of accessing the work step desired by the user and providing a corresponding work step screen.

According to one embodiment, if the operator has only a user terminal on the job site, it is possible to perform the work easily and conveniently without having the work manual directly, without the help of skilled experts for the equipment. In addition, there is no need for a wearable device such as an immersive display that requires separate hardware such as smart glass, and can be used in combination.

By providing remote engineering based on augmented reality, it is possible to provide a more realistic screen, so that a worker can perform work more intuitively. Furthermore, the accuracy of object recognition can be increased by recognizing objects in images using augmented reality-based image analysis and machine learning.

Furthermore, it can be easily and quickly operated even when multi-step operation is required. By using screen touch, voice recognition, biometric recognition, etc., the user can quickly access and jump to a desired work step, for example, real-time processing is possible, and ease of operation is increased because of less troublesome operation.

In addition, for users who want to produce augmented reality-based work guidance information, scripting tools and templates to easily create it can be provided without additional AR manual production costs.

Remote guidance using a user terminal can be extended not only to industrial sites such as factories or buildings, but also to homes and general home appliances. In addition, it can be extended not only to a number of IoT devices used in smart buildings, smart factories, smart homes, etc., but also to legacy devices manufactured before the IoT era without IoT sensing capability.

Furthermore, when an error is predicted in advance for a plurality of devices and an error prediction-based dictionary service is provided, the error of the device is predicted in advance and the device user is informed, and the user possesses the user. Remote guidance can be easily solved in advance by receiving a remote guidance using one user terminal.

1 is a block diagram of a remote guidance system according to an embodiment of the present invention,
2 is a conceptual diagram of remote guidance according to an embodiment of the present invention,
3 is a conceptual diagram of remote guidance according to another embodiment of the present invention,
4 is a configuration diagram of a remote guidance device according to an embodiment of the present invention,
5 is a detailed configuration diagram of the control unit of FIG. 4 according to an embodiment of the present invention;
6 is a detailed configuration diagram of the output unit of FIG. 4 according to an embodiment of the present invention;
7 is a configuration diagram of a remote server according to an embodiment of the present invention,
8 is a flowchart illustrating a remote guidance method according to an embodiment of the present invention,
9 is a diagram illustrating a process in which a remote guidance device according to an embodiment of the present invention requests and receives an augmented reality manual from a remote server;
10 is a view showing a movement of a work step through voice recognition using a voice recognition code according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains. It is provided to fully inform the holder of the scope of the invention, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification.

In the description of the embodiments of the present invention, when it is determined that a detailed description of known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted, and terms to be described below are used in the embodiments of the present invention. The terms are defined in consideration of the function of. It can be changed according to the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

Combinations of each block in the accompanying block diagrams and steps of the flow charts may be performed by computer program instructions (execution engines), these computer program instructions being incorporated into a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment. Since it can be mounted, the instructions executed through a processor of a computer or other programmable data processing equipment create a means to perform the functions described in each block of the block diagram or in each step of the flowchart.

These computer program instructions can also be stored in computer readable or computer readable memory that can be oriented to a computer or other programmable data processing equipment to implement a function in a particular way, so that computer readable or computer readable memory The instructions stored in it are also possible to produce an article of manufacture containing instructions means for performing the functions described in each block of the block diagram or in each step of the flowchart.

And since computer program instructions may be mounted on a computer or other programmable data processing equipment, a series of operation steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer to generate a computer or other programmable It is also possible for instructions to perform data processing equipment to provide steps for executing the functions described in each block of the block diagram and in each step of the flowchart.

In addition, each block or each step can represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical functions, and in some alternative embodiments referred to in blocks or steps It should be noted that it is possible for functions to occur out of sequence. For example, two blocks or steps shown in succession may in fact be executed substantially simultaneously, and it is also possible that the blocks or steps are performed in the reverse order of the corresponding function as necessary.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention exemplified below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. Embodiments of the present invention are provided to more fully describe the present invention to those of ordinary skill in the art.

1 is a block diagram of a remote guidance system according to an embodiment of the present invention.

Referring to FIG. 1, the remote guidance system 1 includes a remote server 2 and a remote guidance device 3. The remote guidance device 3 is included in the user terminal 4 in the form of hardware or software.

The remote guidance system (1) is used for augmented reality and machine learning through a remote guidance device (3) in a user terminal (4) possessed by an operator targeting equipment requiring work at a work site. Based on the work manual in the form of augmented content. The operator can perform the work easily and simply without the help of a skilled expert on the equipment, even without having a work manual. Equipment includes machines, equipment, and devices installed at a work site, and may include components, modules, and the like constituting one equipment. Work includes repair, maintenance, maintenance, adjustment, manipulation, and the like. For example, the work may be to repair the equipment in question, or to take action on equipment that is likely to cause the problem before the problem occurs.

The work site may be an industrial site such as a factory (smart factory) or a building (smart building), or a home (smart home). For example, an industrial site can provide remote guidance for air conditioners, ventilation, air conditioners, fans, boilers, cooling towers, pumps, temperature/humidity sensors, refrigerators, lighting equipment, power equipment, and fire systems in factories. Through the remote server (2), it is possible to comprehensively control and monitor the entire situation of the building, and it is possible to control machinery, lighting, power, access control, disaster prevention, parking management, and facility management. The operator can use the remote guidance information seen through the remote guidance device 3 to smoothly perform work without the help of experts.

The remote guidance system 1 according to another embodiment is operated in a home environment. For example, through a remote server 1, home appliances such as TVs, air conditioners and refrigerators, energy-consuming devices such as water, electricity, air-conditioning, security devices such as door locks and surveillance cameras, and expensive automobiles and electric vehicles. It is remotely adjustable while monitoring and controlling the device. In addition, when a problem occurs in the corresponding home device, if the consumer only has the user terminal 4, the remote guidance for solving the problem can be provided, so that the problem of the device can be easily solved. In this case, the consumer does not need to contact a consumer center to call an expert or separately pay for travel expenses, and waste of time can be prevented. In addition, even if there is no immersive display that requires separate hardware such as smart glass, it can be used in combination.

The remote guidance system 1 may provide remote guidance for a plurality of IoT devices used in smart devices, smart buildings, and smart factories. The IoT device may be a digital signage, a smart TV, a smart watch, a smart CE, and a drone. The remote guidance system 1 according to another embodiment may provide a work guidance targeting a legacy device manufactured before the IoT era without IoT sensing capability. For example, it is possible to provide remote guidance using the user terminal 4 even for legacy devices that cannot sense information about the device and its surroundings because there is no IoT sensor embedded or mounted on the device.

The remote guidance system 1 according to an embodiment predicts an error in advance for a plurality of devices and provides an error prediction-based before service. The dictionary service is a service that predicts errors that may occur to devices in advance, and manages and processes errors before errors occur. For example, when a device error is predicted in advance through the remote server 2, when the device user is notified, the user is provided with a remote guidance using the user terminal 4 owned by the user to easily advance the error. Can be solved.

For an embodiment of augmented reality based work guidance, for example, a worker located at the work site photographs the work site in real time through the camera of the user terminal 4. Then, the remote guidance device 3 receives the live image from the camera and outputs it to the screen. At this time, a virtual work manual that automatically guides the work manual is combined with the screen output image and displayed on the screen through the display of the user terminal 4.

The remote guidance device 3 acquires the current image in the work site, and generates and displays an augmented reality manual (AR manual, hereinafter referred to as the'AR manual') that combines a virtual work manual with the acquired current image. At this time, not only the AR manual, but also virtual operation guidance information including identifier information, location information, status information, work tool information, etc. for the work object may be further displayed on the screen. The identifier information is a virtual object for identifying the work object. For example, it informs the work object with an indicator such as a dotted line or a block. Location information is a virtual object to inform the location of the work object. For example, a virtual finger or the like indicates the location of equipment that needs work or parts of the equipment. The work manual information is a virtual object to inform guide information according to the sequential work manual. The work tool information is a virtual object to inform the tool necessary for the work. For example, it displays all the virtual tools needed for the job, such as a screwdriver and a hammer. An embodiment of the job guidance information displayed on the screen will be described later with reference to FIGS. 2 and 3.

In order to apply augmented reality to a current image photographed in real time, the remote guidance device 3 according to an embodiment repeats a sequential process of object recognition→object tracking→object rendering to link a virtual object to the current image. For example, the remote guidance device 3 recognizes a real object from the current image, tracks the recognized real object, and combines and renders a virtual object corresponding to augmented reality-based work guidance information for the tracked real object, or , After rendering the virtual object and combining it with the real object, it repeats a series of processes to generate the AR manual.

Since the actual image captured through the camera is not fixed, but is continuously moved due to the movement of the operator, it is necessary to continuously follow the movement of the recognized object in the actual image, which is called "object tracking". "Object rendering" is a process of creating a 3D image by injecting a sense of realism in consideration of external information such as a light source, location, and color to a 2D image. Wire frame, ray tracing rendering methods, and the like. The rendering may be performed in the remote guidance device 3, but in the case of a distributed environment, rendering may be performed in the remote server 2 and the execution result may be transmitted to the remote guidance device 3.

The remote guidance system 1 according to an embodiment uses augmented reality and machine learning together when recognizing an object. The core technology of augmented reality is summarized as object recognition, tracking, and rendering, among which object recognition is a method of recognizing an object by an outline. Because the object is recognized through the outline, there is a risk that the recognition rate is lowered when the complex wireframe state is received or when the state of the object is rusted or partially distorted. Therefore, since the recognition accuracy is reduced only by augmented reality, it is complemented by machine learning to increase the object recognition rate. Since machine learning can pass the result values learned through various image databases, it can deliver a more accurate state of recognition for objects that are not identified through contour recognition, and it can be processed by comparing it with the augmented reality module. There is an advantage that can improve the recognition rate. Machine learning may take time depending on the expansion of the field, so first of all, remote guidance is provided through augmented reality method first, and then technical limitations that apply both machine learning and augmented reality are solved, thereby reducing the time constraints of field expansion. can do. In a distributed environment, it is possible to have a negotiation process that increases accuracy by comparing the state of equipment recognized by machine learning in the remote server 2 with the state of equipment recognized by the remote guidance device 3.

In the case of remote guidance processing based on augmented reality and machine learning, AR manual provides procedures for manipulating machines or parts in the field and real-time processing of expert support for workers' understanding of the work. Is provided as. At this time, in the case of a product requiring a large number of manipulations or the difficulty of each step is high, there are many cases in which the steps of a manual with a large number of processing and manipulations are repeated, or a manual operation must be attempted by reversing the manual.

The remote guidance device 3 according to an embodiment uses a process marker capable of grasping the state of a step when multi-step manipulation is required. The process marker is displayed on the screen as each work step identifier of the AR manual is arranged in a line and a marker is recorded in at least one work step identifier. If there are multiple AR manuals, each AR manual can be displayed separately by serial number.

The process marker allows users to access the desired work step based on the current work step. The access to the work step corresponding to the user's operation includes moving the work step in the backward direction based on the current work step, moving the work step in the forward direction, repeating the current work step, and moving to the first work step And moving to other augmented reality manuals. For example, the user can quickly move to the desired position in the reverse direction or forward direction, that is, jump, so that the operation can be easily handled. It is possible to record 5 or 10 regular markers in the work step identifiers and process the user to move to a desired work step using the marker.

The AR manual is separated from the remote server 2 according to the order of the AR manual, and the remote guidance device 3 can request and download the remote server 2 when necessary. If there are multiple AR manuals, they can be classified into serial numbers such as 1001, 1002, 1003, etc., and work steps can be classified into 20 steps, 10 steps, etc. in the serial numbers.

The screen where the AR manual is displayed is a user terminal screen such as a user's tablet PC, a smartphone, and a wearable device having a screen display. At this time, in the case of a tablet PC or a smart phone that can touch the screen, it shows and marks the work step position of the currently displayed AR manual to move the work step in the reverse or forward direction, and displays markers on important work steps before or after. Assign to show the marker location. Among them, a predetermined marker may be designated by a finger touch to approach a work step at a designated marker position to provide a corresponding work step screen. When the user terminal is a wearable device, it is accessible through a predetermined button or the like.

To approach through a user's touch, it is accompanied by inevitable shaking, and it is possible to jump to a predetermined work step through a marker, but it prevents rapid processing because tracking and machine learning of the equipment or parts captured on the display must be performed again. To compensate for this, voice recognition or biometric codes are assigned to important work steps to preserve the work of the operator and to obtain a quick response. In the voice recognition or biometric code, identifier information and location information are assigned to the corresponding code. For example, if the recognized user's voice is "From the beginning again!", for example, the work step approach using the voice recognition code is a work step corresponding to the first work step to which the voice recognition code corresponding to the first position of the work steps is assigned. Provide by moving. If the received user's voice is "5 steps back!", the voice recognition code corresponding to the position after step 5 is moved to the assigned work step based on the current work step. For example, in the case of a work step approach using a biometric code, if the recognized user biometric signal is "3 times in a blink of an eye", it is intended to "restart from the beginning", and the task in which the biometric code corresponding to the initial position is assigned Go to step to provide. If the recognized user biometric signal is "nodding head", "next" is intended, and the biometric code corresponding to the next location is moved to the assigned work step and provided.

The remote guidance system 1 according to an embodiment provides a scripting tool for easily manufacturing a user who wants to produce augmented reality based work guidance information. Accordingly, it is possible to manufacture without additional AR manual production cost. When providing a scripting tool, it is possible to provide a number of templates that can be applied samples for each field. Producers can be small businesses and equipment suppliers. For example, the remote guidance device 3 easily generates augmented reality based work guidance information using a user terminal 4, for example, a template provided as a default in a scripting tool with a desktop. The remote guidance device 3 may receive the augmented reality-based work guidance information produced in this way and use it for generating an AR manual.

The user terminal 4 provided with the remote guidance device 3 includes all portable electronic devices such as a smart phone, a tablet PC, and a wearable device having a screen display. The operator may be provided with work guidance through the remote guidance device 3 without or in conjunction with the need for an immersive display that requires separate hardware, such as smart glass. Of course, if the operator wears an immersive display, it is possible to work in conjunction with it. The user terminal 4 may be wearable equipment.

The remote server 2 monitors equipment located at the work site and performs remote engineering on equipment requiring work. At this time, the remote server 2 may perform remote engineering to solve the error when an error occurs in a predetermined device. As another example, remote engineering can be performed to monitor each piece of equipment on the job site to predict errors in advance and to handle errors for the piece of equipment before an error occurs. Examples of remote engineering include video call, chat, remote screen sharing, and remote control. If the worker cannot solve the problem through the augmented reality-based work guidance information provided through the remote guidance device 3, the remote guidance device 3 connects to the remote server 2 to receive remote control from an expert Can. The remote server 2 may be designed based on a cloud platform in consideration of scalability.

2 is a conceptual diagram of remote guidance according to an embodiment of the present invention.

Referring to FIG. 2, the remote guidance system recognizes an object as a work target in the current image using augmented reality based image analysis of the remote guidance device 3 and machine learning of the remote server 2. To this end, the remote guidance device 3 shares the captured current image with the remote server 2. The remote server 2 recognizes an object in the current image using learning data accumulated through machine learning in advance. The machine learning-based object recognition result is transmitted to the remote guidance device 3, and the recognized object identifier (ID) may be included. For example, the remote server 2 transmits ID1: power panel cover, ID2: breaker, ID3: power supply, which are object identifiers recognized through machine learning in the current image, to the remote guidance device 3. The remote guidance device 3 recognizes the outline of the object in the current image through augmented reality-based image analysis, and reflects the machine learning-based object recognition result of the remote server 2 in the augmented reality-based recognition result, thereby recognizing the accuracy of object recognition. To increase.

The remote guidance device 3 displays the AR manual on the screen by combining the work guidance information with the current image including the recognized object. For example, as shown in FIG. 2, when a problem occurs in the power supply of the power supply in the power panel, the remote guidance device screen displays, "The power of the power supply in the power panel has exceeded a threshold value, so the power must be reduced." Status information 102 such as a warning message is displayed. In addition, the identification information 100 such as an identifier such as a block type is displayed on the power panel cover where the problem occurs, so that the power panel cover is recognized by the operator. The target of the identification information 100 may be changed according to the operation manual.

In addition, the operation manual 104 is also displayed on the screen. For example, a preset work manual is displayed on the screen according to the work order, such as "1. Open the power panel cover. 2. Lower the breaker. 3. Replace the power supply. 4. Raise the breaker." In "1. Open the power panel cover.", the handle of the power panel cover in the current image may be identifiable by a dotted line. In "2. Lower the breaker.", the breaker in the current image may be identifiable by a dotted line or the like. In “3. Replace the power supply.”, the power supply in the current image may be identifiable by a dotted line or the like.

The above-described operation guidance information may be produced in advance through a scripting tool 106 that is dynamically editable according to the operation manual based on a predefined template. If the worker cannot solve the problem using the augmented reality-based work guidance information, the remote guidance device 3 may be networked with the remote server 2 to receive remote engineering support through the remote management 108 from a specialist. . Remote engineering includes video calls, chat, remote screen sharing, and remote control.

3 is a conceptual diagram of remote guidance according to another embodiment of the present invention.

Referring to FIG. 3, when the operation target is an engine, the remote server 2 transmits the machine learning-based object recognition result to the remote guidance device 3. For example, the remote server 2 transmits ID1: radiator fan, ID2: timing belt, and ID3: spark plug, which are object identifiers recognized through machine learning in the current image, to the remote guidance device 3. The remote guidance device 3 recognizes the outline of the object in the current image through augmented reality-based image analysis, and reflects the machine learning-based object recognition result of the remote server 2 in the augmented reality-based recognition result, thereby recognizing the accuracy of object recognition. To increase.

The remote guidance device 3 displays the AR manual on the screen by combining the work guidance information with the current image including the recognized object. For example, as illustrated in FIG. 3, when a problem occurs with the engine, the identification information 100, such as an identifier such as a square shape of a dotted line, is displayed on the screen for the engine to recognize to the operator that the engine is a work target. Order. The target of the identification information 100 may be changed according to the operation manual.

The operation manual 104 is also displayed on the screen. For example, "1. Open the bonnet. 2. Check the timing belt tension. 3. Check the radiator fan rotation. 4. Check the spark plug." Is displayed on. After the operator opens the bonnet, the timing belt in the current image may be displayed identifiably by a dotted line or the like. After the operator confirms the tension of the timing belt, the radiator in the current image may be displayed identifiably by a dotted line or the like. After the operator confirms the rotation of the radiator fan, the spark plug in the current image may be identifiable by a dotted line or the like.

The above-described operation guidance information may be produced in advance through a scripting tool 106 that is dynamically editable according to the operation manual based on a predefined template. For example, the remote guidance device 3 may allow the user to create work guidance information according to the work manual using the scripting tool 106 through the user terminal 4 such as a desktop. The scripting tool 106 is a tool for the user to easily create augmented reality-based work guidance information, and enables the user to produce it without additional AR manual production cost. When providing the scripting tool 106, it is possible to provide a number of templates that can be applied samples for each field. Users can be small businesses and equipment suppliers. For example, a user can easily create augmented reality-based work guidance information using a template provided as a default in a scripting tool with a remote guidance device 3. The remote guidance device 3 may receive the augmented reality-based work guidance information produced in this way and use it for generating an AR manual. In the template, a template drawing 40 schematically illustrating a virtual control area may be displayed on the control screen. A graphic object representing a virtual tool may be arranged in the template drawing.

If the worker cannot solve the problem using the augmented reality-based work guidance information, the remote server 2 may be networked to receive remote engineering support through the remote management 108 from an expert. Remote engineering includes video calls, chat, remote screen sharing, and remote control.

4 is a configuration diagram of a remote guidance device according to an embodiment of the present invention.

1 and 4, the remote guidance device 3 includes an input unit 31, a control unit 32, an output unit 33, a communication unit 34, and a memory 35.

The input unit 31 receives image data including a current image taken in real time of the work site from the current point of view of the operator, and receives a user manipulation signal if necessary. The user manipulation signal may include at least one of a user's screen touch signal, a key input signal, a voice signal, and a bio signal. The input unit 31 receives image data and a user manipulation signal and transmits it to the control unit 32.

The input unit 31 according to an embodiment includes an image input unit 310, a touch input unit 312, an audio input unit 314, and a bio-input unit 316. The image input unit 310 receives an image of a work site in real time. The touch input unit 312 receives a screen touch signal from a user. The voice input unit 314 is provided with a microphone that receives voice from a user to receive voice signals. The biometric input unit 316 receives a biometric image photographed for a user. The image input unit 310 and the bio-input unit 316 may receive an image from a camera, an image sensor, or the like.

The output unit 33 outputs information processed through the control unit 32. The output unit 33 according to an embodiment outputs the AR manual generated through the control unit 32. As an example of output, an image may be displayed on a screen, and audio may be output through a speaker.

The output unit 33 according to an embodiment displays a process marker on the screen. In the process marker, for each AR manual consisting of a plurality of work steps, each work step identifier is arranged in a line, and a marker is recorded in at least one work step identifier. If there are multiple AR manuals, each AR manual is displayed by serial number. An embodiment of the process marker will be described later with reference to FIG. 10.

The communication unit 34 is connected to a network with the remote server 2 to perform wired/wireless communication. The communication unit 34 interacts with the remote server 2 to maintain the homeostasis of the network. The remote guidance device 3 may be connected to the remote server 2 through a dedicated network through the communication unit 34 or through a general-purpose network such as the Internet.

The communication unit 34 according to an embodiment requests and receives a work manual from the remote server 2. The communication unit 34 may start a work guidance upon receiving a work command from the remote server 2. For example, when the remote server 2 is monitoring equipment in the work site and it detects an error or a possibility of an error, the operation request command for the corresponding equipment is transmitted to the remote guidance device 3 Inform the worker.

The communication unit 34 according to an embodiment transmits and shares the current image acquired through the image input unit 30 to the remote server 2. Then, the object information recognized through the augmented reality based image analysis of the remote guidance device 3 is transmitted to the remote server, and the object information recognized through machine learning can be received from the remote server 2.

The communication unit 34 according to an embodiment supports short-range wireless communication. For example, the communication unit 34 is short-range wireless communication (Near Field Communication: NFC, hereinafter referred to as NFC), RF ID (radio frequency identification: RFID, hereinafter referred to as RFID), magnetic source transmission (Magnetic Source Transmission, hereinafter MST) Support). The communication unit 34 may transmit and receive radio signals by performing short-range wireless communication in the frequency band of several tens to tens of MHz. When a process marker in which a main work step is recorded is provided on a work object, for example, a work equipment, for example, when attached, the communication unit 34 transmits and receives a radio signal through short-range wireless communication with the process marker.

The memory 35 stores data. The data may be unique information of the equipment on the job site, location information, drawings of the equipment, work manual information, parts information in the equipment, work schedule information, and the like. This information may be downloaded from the remote server 2 and stored, or may be input from the user through the input unit 31. The memory 35 is a built-in memory such as a hard disk and a flash memory, as well as a CF card (Compact Flash Card), SD card (Secure Digital Card), SM card (Smart Media Card), MMC As a memory when moving, such as a multi-media card or a memory stick, it may be implemented inside the remote guidance device 3 or may be provided in a separate device.

The control unit 32 controls the operation of each configuration described above. The control unit 32 according to an embodiment repeatedly performs object recognition, tracking, and rendering on the current situation image acquired through the image input unit 30. For example, virtual work guidance information to help work on a work target while recognizing the work target in the current image and tracking the recognized work target is matched to the current image and rendered, or virtual work guidance information is rendered. After that, the AR manual is generated by matching the current video. The control unit 32 according to an embodiment processes a command for remote control of the remote server 2. For example, the screen of the remote guidance device 3 is shared with the remote server 2.

The control unit 32 according to an embodiment uses augmented reality based image analysis and learning data based machine learning for object recognition. The image analysis may use a general algorithm for recognizing objects in the acquired current image. For example, an object is recognized using point cloud information, depth information, and color information in an image. Object analysis-based object recognition is a well-known technique, and thus detailed description is omitted. At this time, an object recognition rate may be increased by processing image processing, for example, binarization of an image, edge blending, and masking.

Machine learning is one of the methods used in artificial intelligence, and it is a technique of learning by inputting a lot of data into a computer and classifying similar ones. For example, if a picture similar to a stored dog picture is input, the computer classifies it as a dog picture. Objects in the image can be recognized using the learning data accumulated through machine learning. The controller 32 intends to increase the accuracy of object recognition by using object recognition using machine learning as an auxiliary means of augmented reality-based image analysis. Machine learning can use SVM (Support Vector Machines), deep learning techniques, and the like.

The control unit 32 according to an embodiment provides a corresponding work step screen by approaching a work step corresponding to a user operation among a plurality of work steps constituting an AR manual according to a user manipulation signal. At this time, the user manipulation signal includes a user's screen touch signal, audio signal, video signal, and bio signal. The access to the work step corresponding to the user's operation can be performed by moving the work step in the reverse direction based on the current work step, moving the work step in the forward direction, repeating the current work step, moving to the first work step, and other augmented reality manuals. And the like. The detailed configuration of the control unit 32 will be described later with reference to FIG. 5.

When the remote guidance device 3 is connected to the remote server 2, the remote guidance device 3 becomes an agent that performs control commands received from the remote server 2. The remote guidance device 3 may run an application for remote guidance. At this time, it has a form capable of supporting all different OSs such as different OSs, for example, Android, iOS, and Windows.

5 is a detailed configuration diagram of the control unit of FIG. 4 according to an embodiment of the present invention.

1, 4 and 5, the control unit 32 includes a signal processing unit 320, a remote control processing unit 324 and a device control unit 326. The signal processing unit 320 may include an image recognition unit 3200, a touch recognition unit 3202, a voice recognition unit 3204, a raw vegetable recognition unit 3206, and a wireless signal recognition unit 3208.

The image recognition unit 3200 recognizes an object in the current image using augmented reality-based image analysis of the current image and machine learning using learning data if necessary. For example, the image recognition unit 3200 increases the object recognition accuracy by reflecting the machine learning-based object recognition result of the remote server 2 received through the communication unit 34 to the augmented reality-based object recognition result. The image recognition unit 3200 tracks the movement of the recognized object. Since the current image captured through the camera is not fixed, but continues to move due to the movement of the operator, the image recognition unit 3200 continues to follow the movement of the recognized object within the current image. Subsequently, the image recognition unit 3200 calculates the interaction between the real environment and the virtual environment and reflects the calculation result to generate an AR manual. The AR manual is a combination of a virtual environment and a real environment.

The image recognition unit 3200 according to an embodiment calculates the interaction between the real object and the virtual object and reflects the calculation result, combines the virtual object with the current image and renders it, or renders the virtual object and renders it to the current image. Combine to create an AR manual. At this time, the location of the virtual object to represent the operation guidance information in the current image can be selected and the operation guidance information can be arranged according to the operation manual at the selected location of the virtual object.

The image recognition unit 3200 according to an embodiment receives pre-made work guidance information through a scripting tool that can be dynamically processed according to a work manual based on a predefined template and uses it for generating an AR manual. The AR manual generated through the image recognition unit 3200 is provided to the output unit 33.

The touch recognition unit 3202, the voice recognition unit 3204, the raw vegetable recognition unit 3206, and the wireless signal recognition unit 3208 are modules for processing a hypermotion step.

When the touch input unit 312 receives a screen touch signal from a user, the touch recognition unit 3202 receives a user touch signal from the touch input unit for a process marker displayed on the screen, recognizes the user touch, and configures an AR manual The work step corresponding to the recognized touch among the work steps is approached to provide a corresponding work step screen.

The voice recognition unit 3204 assigns a voice recognition code having identifier information and location information to work steps constituting the AR manual. When the voice input unit 314 receives the voice from the user through the microphone, the voice recognition unit 3204 receives the user voice signal from the voice input unit 314 to recognize the user voice and corresponds to the recognized voice among a plurality of work steps The voice recognition code to be accessed is approached to the assigned work step and a corresponding work step screen is provided.

The biometric recognition unit 3206 allocates a biometric code having identifier information and location information to work steps constituting the AR manual. When the biometric input unit 316 receives a biometric image photographed for a user, the biometric recognition unit 3206 receives the user biometric image from the biometric input unit 316, detects the user's face and body, and detects the detected face and body The facial expression or motion is recognized from the information, and a biometric code corresponding to the recognized facial expression or motion is accessed from a plurality of work stages, and a corresponding work stage screen is provided. The user's face and body detection may include the user's eye blinks, irises, fingerprints, hand movements such as waving hands, finger shapes such as two fingers spreading, and head movements such as nodding heads. Can.

The wireless signal recognition unit 3208 recognizes a work target, for example, a process marker provided in the work equipment, by the wireless signal received from the communication unit 34. The key markers for the work object are recorded in the process marker, so that the key task steps recorded in the recognized process marker can be easily identified. Further, among the identified major work steps, the user may approach the work step desired by the user and provide a corresponding work step screen. Wireless communication can use NFC, RFID, MST, and the like.

The remote control processing unit 324 requests remote control from the remote server 2 and remotely shares the screen under the control of the remote server 2. The device control unit 326 processes the control request of the remote server 2.

6 is a detailed configuration diagram of an output unit of FIG. 4 according to an embodiment of the present invention.

Referring to FIG. 6, the output unit 33 includes an identifier display unit 330, a status display unit 332, a work manual display unit 334, a work tool display unit 336, and a process marker display unit 338.

The identifier display unit 330 displays a virtual object for identifying a work object on the screen. For example, identifiers such as arrows, lines, and blocks are displayed on the screen. At this time, the work object may be changed according to the work manual. Since the real screen shot in real time continues to move, the identifier also moves according to the moving work object. The identifier display unit 330 may inform the location of the work object. For example, a hand shape indicates a part that needs work.

The status display unit 332 displays various status information about the equipment, such as the control status of the work target, a failure status, an operation status, time information, and control parameters required for work in the equipment. At this time, a detailed description may be displayed on the screen, including the part number of the work object, life cycle, recent replacement history, and failure history. As an example of the control parameter, when the equipment is an air conditioner, the control parameter may be a target temperature (or set temperature), an upper limit temperature, a lower limit temperature, an air volume, operation, stop or the like. If the equipment is a lighting fixture, the control parameter may be lighting brightness or the like. The status display unit 332 may automatically display status information for equipment requiring work, but may also display status information at the request of an operator. For example, when an operator selects a predetermined equipment in the screen, various status information on the selected equipment can be displayed on the screen.

The work manual display unit 334 displays a virtual object for informing a sequential work manual on the screen. At this time, it is possible to inform the operator of the equipment or parts in the equipment that need work in accordance with the operation manual through the identifier display unit 330. The work tool display unit 336 displays a virtual object for informing a tool necessary for work on the screen. Work tools include, for example, hammers, screwdrivers, and hands.

The process marker display unit 338 displays a process marker on the screen so as to grasp the state of the step targeting the AR manual composed of a plurality of work steps. The process marker includes identifier information per AR manual, identifier information for each work step in the AR manual, and marker information assigned to at least one work step identifier based on the current work step for access to a work step desired by the user do.

In the case of speech recognition, the process marker display unit 338 assigns speech recognition codes having identifier information and location information to the work steps constituting the AR manual. In the case of biometric recognition, the process marker display unit 338 assigns biometric codes having identifier information and location information to work steps constituting the AR manual.

7 is a block diagram of a remote server according to an embodiment of the present invention.

1 and 7, the remote server 2 includes an authentication management module (VMM) 20, a monitoring screen module (MSM) 21, and a network handling module (NHM). (22), Remote Control (Remote Access & Control Module: RCM) 23, Data Processing Module (DPM) 24, Client Management (Device & App Monitoring Module: DMM) 25, Protocol Processing ( It includes a Messaging Protocol Stack (MPS) 26 and a Big Data Statistics Engine (BSE) 27. The remote server 2 may be designed based on a cloud platform in consideration of scalability.

The authentication management unit 20 verifies the validity of user information of the requested worker and authenticates who is the subject of the requirements received by the remote server 2 as necessary. The monitoring unit 21 monitors the screen of the remote guidance device 3 of the remote operator. At this time, it provides functions to monitor or manipulate through the terminals of technicians and managers.

The network connection unit 22 maintains the homeostasis of the network connection regardless of the network type. The network connection unit 22 transmits and receives information to and from the operator's remote guidance device 3. For example, the network connection unit 22 receives and shares the current image from the remote guidance device 3, receives object information recognized through image analysis based on augmented reality of the remote guidance device 3, and learns the machine. The base object recognition result is transmitted to the remote guidance device 3. When the remote control unit 23 receives the remote control request from the remote guidance device 3, it shares the screen of the remote guidance device 3 and transmits a remote control command to the remote guidance device 3. At this time, the necessary process adjustment function is provided.

The data processing unit 24 recognizes the object through machine learning based on the learning data on the current image received from the remote guidance device 3 through the network connection unit 22. The client management unit 25 checks the status of the client and, if the client operates normally or not, identifies and processes any problems. The client may be a user terminal carried by an operator, an application executed in the remote guidance device 3, or various types of equipment in a work site. The remote guidance device 3 combines and provides augmented reality-based work guidance information to the current image captured by the operator. The client management unit 25 according to an embodiment monitors equipment in a work site to predict an error in advance and generates a work command for handling an error for the corresponding equipment before an error occurs. The generated work command is transmitted to the remote guidance device 3 of the worker through the network connection 22.

The protocol processing unit 26 is a software module that receives a client's normal signal, parses it, and implements a communication protocol for processing it. The data analysis unit 27 analyzes the data received from the client so as to grasp the abnormal signs for a plurality of clients and provide a stable service. At this time, anomalies may be identified by comparing the analyzed data with a preset threshold value.

8 is a flowchart illustrating a remote guidance method according to an embodiment of the present invention.

Referring to FIG. 8, the remote guidance device acquires (800) the current image of the work site in real time from the current point of view of the operator, recognizes the work object from the acquired current image, and tracks the recognized work object (810). . Subsequently, an operation manual for guiding the operation of the operator with respect to the tracked work object is acquired (820 ). The AR manual is provided to the user by matching the received virtual work manual to the current image including the tracked work target (830). Subsequently, a user operation signal is received, the user operation is recognized, and a work step screen corresponding to the recognized user operation is approached among a plurality of work steps constituting the AR manual to provide a corresponding work step screen (840 ).

In the work step access and provision step 840. The remote guidance device according to an embodiment receives a touch signal from a user targeting a process marker displayed on a screen, recognizes a user touch, and approaches a work step corresponding to the recognized touch among a plurality of work steps constituting an AR manual. Provides the corresponding work step screen. Further comprising the step of assigning a voice recognition code having identifier information and location information to the work steps constituting the AR manual. In this case, a user may recognize a user voice by receiving a voice signal from the user, and access to a work step to which a voice recognition code corresponding to the recognized voice is assigned among a plurality of work steps to provide a corresponding work step screen.

In the operation step approaching and providing step 840, the remote guidance device according to an embodiment displays a process marker on the screen for an AR manual composed of a plurality of operation steps. The process marker includes identifier information per AR manual, identifier information for each work step in the AR manual, and marker information assigned to at least one work step identifier based on the current work step for access to a work step desired by the user do.

In the operation step access and provision step 840, the remote guidance device according to an embodiment allocates a biometric code having identifier information and location information to the operation steps constituting the AR manual. In addition, a biometric image corresponding to a recognized facial expression or motion among a plurality of work steps is recognized by receiving a biometric image photographed for a user, detecting a user's face and body, and recognizing facial expressions and motions from the detected face and body information. You can access the assigned work step and provide the corresponding work step screen.

The remote guidance device according to an embodiment may recognize the process marker by the received radio signal by performing short-range wireless communication with the process marker provided in the work object and grasp the work step recorded in the recognized process marker. Further, among the identified major work steps, the user may approach the work step desired by the user and provide a corresponding work step screen.

9 is a diagram illustrating a process in which a remote guidance device according to an embodiment of the present invention requests and receives an AR manual from a remote server.

Referring to FIG. 9, the remote guidance device 3 requests and receives an AR manual from the remote server 2. AR manuals can be identified by serial number. For example, the remote guidance device 3 downloads the AR manual 1001 from the remote server 2 (900) and displays the AR manual 1001 on the screen. Each AR manual can be divided into multiple work stages, and the AR manual 1001 is divided into 10 stages. Subsequently, the remote guidance device 3 requests the AR manual 1002 from the remote server 2 (910), downloads the AR manual 1002 from the remote server 2 (920), and follows the AR manual 1001 to display the AR manual 1002 on the screen. Display.

10 is a view showing a movement of a work step through voice recognition using a voice recognition code according to an embodiment of the present invention.

Referring to FIG. 10, the remote guidance device 3 according to an embodiment uses a process marker capable of grasping the state of a step when a multi-step operation is required. The process marker is displayed on the screen as each work step identifier of the AR manual is arranged in a line and a marker is recorded in at least one work step identifier. If there are multiple AR manuals, each AR manual can be displayed separately by serial number.

Using the process marker, the user can quickly move to a desired position in a backward or forward direction based on the current work step, thereby facilitating the work. For example, 5 or 10 regular markers are recorded in the work step identifiers, and the user is processed to move to a desired work step using the marker. It is possible not only to move to the desired front-rear work step, but also to repeat the current work step.

The AR manual is separated from the remote server 2 according to the order of the AR manual, and the remote guidance device 3 can request and download the remote server 2 when necessary. If there are multiple AR manuals, they can be classified into serial numbers such as 1001, 1002, 1003, etc., and work steps can be classified into 20 steps, 10 steps, etc. in the serial numbers.

To approach through a user's touch, it is accompanied by inevitable shaking, and it is possible to jump to a predetermined work step through a marker, but it prevents rapid processing because tracking and machine learning of the equipment or parts captured on the display must be performed again. To compensate for this, voice recognition or biometric codes are assigned to important work steps to preserve the work of the operator and to obtain a quick response. In the voice recognition or biometric code, identifier information and location information are assigned to the corresponding code.

For example, a work step approach using voice recognition code, the present work step (Present Marker) and the current work step (Present Marker) based on the previous and subsequent major steps (Marker 1, Marker 2, Marker 4) Is marked with a marker. At this time, if the recognized user voice is "1001 again!", the voice recognition code corresponding to the marker 1 position of the AR manual 1001 is moved to the first assigned operation step and provided. If the received user voice is "back!", the voice recognition code corresponding to the marker 2 position displayed on the basis of the current work step is moved to the assigned work step and provided. If the received user voice is "forward!", the voice recognition code corresponding to the marker 4 position displayed in front of the current work step is moved to the assigned work step and provided. If the received user voice is "AR manual 1001!", the voice recognition code corresponding to the marker 1 position of the AR manual 1001 is moved to the assigned work step and provided. If the received user voice is "1002 again!", the voice recognition code corresponding to the marker 5 position of the AR manual 1002 is moved to the assigned work step and provided.

For example, in the case of a work step approach using a biometric code, if the recognized user biometric signal is "3 times in a blink of an eye", it is intended to "restart from the beginning", and the task in which the biometric code corresponding to the initial position is assigned Go to step to provide. If the recognized user biometric signal is "nodding head", "next" is intended, and the biometric code corresponding to the next location is moved to the assigned work step and provided.

So far, the present invention has been focused on the embodiments. Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in terms of explanation, not limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be interpreted as being included in the present invention.

Claims (16)

In the remote guidance device provided in a portable user terminal that does not need to wear an immersive display,
An input unit that receives a current image of a worker's field situation;
Recognizes the work object from the current image received from the input unit, matches the virtual work manual to the current image including the recognized work object, provides the augmented reality manual, and provides identifier information and location information in the work steps constituting the augmented reality manual. Assigning a voice recognition code having a, to recognize a user voice instructing access to a predetermined work step by receiving a user voice signal, and a work step in which a voice recognition code corresponding to the recognized voice is assigned among a plurality of work steps. A control unit that accesses and provides a corresponding work step screen; And
An output unit for arranging and displaying each work step of the augmented reality manual in a line, and displaying a process marker on at least one work step based on a current work step for access to a work step desired by a user;
A remote guidance device comprising a. delete According to claim 1,
The access to the work step corresponding to the user's voice can be performed by moving the work step in the reverse direction based on the current work step, moving the work step in the forward direction, repeating the current work step, moving to the first work step or other augmented reality manual. Remote guidance device comprising a movement of the. According to claim 1, The output unit
A process marker that can identify the status of a step is displayed on the screen targeting the augmented reality manual composed of a number of work steps,
The process marker includes identifier information for each augmented reality manual, identifier information for each work step in the augmented reality manual, and marker information assigned to at least one work step identifier based on the current work step for access to a work step desired by the user A remote guidance device comprising a. delete delete delete The method of claim 1, wherein the input unit
An image input unit for acquiring an image of a current situation photographing a work site in real time; It includes,
The control
Based on augmented reality, it recognizes an object from the current image, tracks the movement of the recognized object, and combines and renders virtual objects corresponding to the augmented reality-based work guidance information for the tracked real objects or renders virtual objects An image recognition unit that combines them with real objects to generate an augmented reality manual;
A remote guidance device comprising a. The method of claim 8, wherein the remote guidance device
It is networked with a remote server and transmits and shares the current image acquired through the input unit to the remote server, transmits the object information recognized through the augmented reality based image analysis of the remote guidance device to the remote server, and recognizes it through machine learning from the remote server Communication unit for receiving the object information; Further comprising,
The image recognition unit
A remote guidance device characterized by recognizing a contour recognition result of an augmented reality based object by recognizing a machine learning based object recognition result of a remote server received from a communication unit, and recognizing an unknown object when recognizing the contour of the augmented reality based object. The method of claim 1, wherein the remote guidance device
A communication unit capable of short-range wireless communication and a process marker provided on the work object; Further comprising,
The control
A radio that recognizes a process marker by a radio signal received from the communication unit, identifies the work step recorded in the recognized process marker, and provides a corresponding work step screen by accessing the work step desired by the user by user manipulation during the identified work step Signal recognition unit;
A remote guidance device comprising a. In a remote guidance method using a remote guidance device provided in a portable user terminal that does not need to wear an immersive display, the remote guidance device includes:
Acquiring a current image of a worker's field situation, recognizing a work object from the current image, and tracking the recognized work object;
Obtaining a work manual for guiding a worker's work on the tracked work object;
Providing an augmented reality manual by matching the received virtual work manual to the current image including the tracked work target;
Assigning a voice recognition code having identifier information and location information to work steps constituting the augmented reality manual and arranging and displaying each work step in a line;
Displaying a process marker on at least one work step based on a current work step for access to a work step desired by a user; And
Recognizing a user's voice instructing access to a predetermined work step by receiving a user voice signal, and accessing a work step to which a voice recognition code corresponding to the recognized voice is assigned among a plurality of work steps to provide a corresponding work step screen step;
Remote guidance providing method comprising a. The method of claim 11,
The process marker includes identifier information for each augmented reality manual, identifier information for each work step in the augmented reality manual, and marker information assigned to at least one work step identifier based on the current work step for access to a work step desired by the user Remote guidance providing method comprising a. delete delete delete The method of claim 11, wherein the method for providing remote guidance
Short-range wireless communication with the process marker provided on the work object; And
The process marker is recognized by the wireless signal received through wireless communication, the work step recorded in the recognized process marker is identified, and the user's operation accesses the work step desired by the user during the identified work step to provide the corresponding work step screen To do;
A method for providing remote guidance, further comprising a.

Images