KR20190056935A - Mobile terminal providing augmented reality based maintenance guidance, remote managing apparatus and method for remote guidance using the same - Google Patents

Abstract

Disclosed are a mobile terminal, a remote managing apparatus, and a method for remote guidance based on augmented reality using the mobile terminal and the remote managing apparatus. According to one embodiment of the present invention, the mobile terminal comprises: an image obtaining unit obtaining a real image photographed at the present point of a worker; a processor creating augmented reality content by matching virtual work guidance information to recognize target work in the obtained real image, track the recognized target work, and help work for the target work to the real image including the tracked target work; and an output unit outputting the generated augmented reality content.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mobile terminal, a remote management apparatus, and an augmented reality-based remote guidance method using the same,

The present invention relates to augmented reality-based image analysis and remote control techniques.

Maintenance of equipment requires many years of experience, and it takes a long time to master it when new equipment is introduced even by skilled users. Also, no matter how skilled, it is impossible to know all maintenance procedures, and it is difficult to predict when and what maintenance needs will occur. As a result, mechanics have always carried the manual for equipment maintenance and maintenance work.

Maintenance manuals are usually made in books or 2D images. The problem with the manual made in the book is that if one of the parts fails, the corresponding page must be searched one by one, and if the manual is outdated, the color is damaged or the damage becomes worse. In the case of complex types of mechanical equipment such as airplanes, hundreds of thousands of manuals are required, so the amount is too large to carry.

Such maintenance and other problems are not limited to industrial sites. For example, in the case of a refrigerator in a home, consumers often can not fix themselves, so it is often necessary to call a specialist and repair it. In this case, it takes a lot of time and money to solve the problem, which is inefficient.

According to one embodiment, a portable terminal and a remote management apparatus that can easily and quickly work according to a sequential instruction provided on the basis of an augmented reality even if the apparatus is of a complicated structure or is not an experienced operator, and an augmented reality- We propose a guidance method.

The portable terminal according to an embodiment includes an image acquiring unit that acquires an actual image photographed at a current point of time of an operator, and an image acquiring unit that recognizes an image of the object in the acquired real image, A processor for generating augmented reality content by matching the virtual work guidance information for the actual work with the actual image including the tracked work object, and an output unit for outputting the generated augmented reality contents.

The processor includes an object recognition unit that recognizes an object in an actual image using augmented reality-based image analysis and machine learning, a tracking unit that tracks the movement of the recognized object, An augmented reality processing unit for providing the operation guidance information based on the augmented reality by generating the augmented reality contents combining the virtual environment with the actual environment by calculating the action and reflecting the calculation result, And a visualization unit provided to the output unit.

The portable terminal communicates with the remote management apparatus via a network to share the actual image acquired through the image acquisition unit with the remote management apparatus and transmits the recognized object information to the remote management apparatus through the augmented reality based image analysis of the portable terminal Wherein the object recognition unit reflects the result of recognition of the object based on the machine learning of the remote management apparatus received through the communication unit to the object recognition result based on the augmented reality, Can be recognized.

The augmented reality processing unit calculates the interaction between the actual object and the virtual object, combines the virtual object with the actual image, reflects the calculation result, and renders or renders the virtual object, and then combines the virtual object with the actual image to generate the augmented reality content, It is possible to select a virtual object position to express the work guidance information in the real image and arrange the work guidance information in accordance with the work order at the selected virtual object position.

The augmented reality processing unit may receive the work guidance information generated in advance through a scripting tool capable of dynamically processing according to a work manual based on a predefined template, and use the generated work guidance information to generate the augmented reality contents.

The processor may further include a remote control processing unit for requesting the remote management apparatus to perform remote control and sharing the portable terminal screen remotely according to the remote control of the remote management apparatus, and a device control unit for processing the control request of the remote management apparatus .

The output unit includes an identifier display unit for displaying a virtual object for identifying a work object on a screen, a status display unit for displaying a virtual object for notifying the status of the work object on the screen, and a virtual object for guiding a sequential work order to the screen And a work tool display unit for displaying on the screen a virtual object informing the tool necessary for the work.

A remote management apparatus according to another embodiment of the present invention includes a client management unit for managing a mobile terminal that combines augmented reality based job guidance information with actual images captured by an operator at a work site, A network connection unit receiving the recognized object information through image analysis based on the augmented reality of the mobile terminal and transmitting the machine learning based object recognition result to the mobile terminal, And a data processing unit for recognizing the object.

The client management unit monitors the equipment in the worksite to predict an error in advance and generates a job command for processing an error for the device before an error occurs. The network connection unit transmits the generated job command to the mobile terminal .

The remote management apparatus may further include a remote control unit that receives a remote control request from the portable terminal, remotely shares a screen of the portable terminal, and transmits a remote control command to the portable terminal to perform remote engineering.

The remote management device may operate based on the cloud.

According to another aspect of the present invention, there is provided a remote guidance method using a mobile terminal, including: obtaining an actual image photographed at a current point of time of a mobile terminal; recognizing an object of operation in the obtained actual image; Generating augmented reality content by matching virtual job guidance information for assisting an operation on an object to be processed with an actual image including a tracked object to be worked on; And outputting.

The step of recognizing the object of operation includes the steps of: sharing the photographed actual image with a network-connected remote management apparatus; firstly recognizing an actual image object through augmented reality-based image analysis; And secondarily recognizing the object by recognizing the object recognition result through machine learning in the remote management apparatus on the object recognition result through the augmented reality based image analysis.

The second recognition of the object includes transmitting the recognized object information to the remote management apparatus through the augmented reality-based image analysis, receiving the object information recognized through the machine learning from the remote management apparatus, Comparing the object information received from the device with the object information recognized by the mobile terminal, and requesting the remote management device to perform object recognition again if the object information is different; receiving a machine learning based object recognition result from the remote management device And modifying the object recognition result of the mobile terminal by reflecting the received object recognition result.

According to one embodiment, if a worker has only a portable terminal at a work site, the work can be easily and easily performed without having to have a work manual directly without the help of a skilled expert. It also eliminates the need for immersive displays that require traditional, separate hardware, such as smart glass.

By providing remote engineering based on Augmented Reality, more realistic images can be provided, allowing operators to work more intuitively. Furthermore, by using augmented reality - based image analysis and machine learning, it is possible to improve object recognition accuracy by recognizing objects in an image.

Also, a scripting tool and a template for easily creating augmented reality-based job guidance information can be produced for a user, thereby making it possible to manufacture without additional augmented reality manual production cost.

Remote guidance using mobile terminals can be extended not only to industrial sites such as factories and buildings, but also to households and general household appliances. In addition, not only a number of IoT devices used in a smart building, a smart factory, a smart home, etc., but also a legacy device manufactured before the IoT era without IoT sensing capability can be extended.

Further, in the case of predicting an error for a plurality of devices and providing an error prediction based prior service, if an error of the device is predicted in advance and informed to the user of the device, It is possible to easily solve the error in advance by receiving the remote guidance using one mobile terminal.

1 is a configuration diagram of a remote guidance system according to an embodiment of the present invention;
2 is a configuration diagram of a remote guidance system for explaining the concept of a remote guidance according to an embodiment of the present invention;
3 is a configuration diagram of a remote guidance system for explaining the concept of remote guidance according to another embodiment of the present invention;
4 is a configuration diagram of a portable terminal according to an embodiment of the present invention;
FIG. 5 is a detailed configuration diagram of the processor of FIG. 4 according to an embodiment of the present invention;
FIG. 6 is a detailed configuration diagram of the output unit of FIG. 4 according to an embodiment of the present invention;
7 is a configuration diagram of a remote management apparatus according to an embodiment of the present invention;
FIG. 8 is a flowchart illustrating an object recognition process through cooperation between a remote management apparatus and a mobile terminal according to an embodiment of the present invention;
FIG. 9 is a portable terminal screen for illustrating an example in which job guidance information is displayed on a portable terminal according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. , Which may vary depending on the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification.

Each block of the accompanying block diagrams and combinations of steps of the flowcharts may be performed by computer program instructions (execution engines), which may be stored in a general-purpose computer, special purpose computer, or other processor of a programmable data processing apparatus The instructions that are executed through the processor of the computer or other programmable data processing equipment will generate means for performing the functions described in each block or flowchart of the block diagram.

These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce an article of manufacture containing instruction means for performing the functions described in each block or flowchart of the flowchart.

And computer program instructions may be loaded onto a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the data processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or step may represent a portion of a module, segment, or code that includes one or more executable instructions for executing the specified logical functions, and in some alternative embodiments, It should be noted that functions may occur out of order. For example, two successive blocks or steps may actually be performed substantially concurrently, and it is also possible that the blocks or steps are performed in the reverse order of the function as needed.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the following embodiments of the present invention may be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to enable those skilled in the art to more fully understand the present invention.

1 is a configuration diagram of a remote guidance system according to an embodiment of the present invention.

Referring to FIG. 1, a remote guidance system 1 includes a remote management apparatus 2 and a mobile terminal 3. The remote guidance system 1 guides the work order based on the augmented reality through the portable terminal 3 held by the worker on the equipment requiring work at the work site. Thus, the operator can easily and easily carry out the work without having to know the work manual, without the help of skilled experts in the equipment. Here, the equipment includes a machine, a facility, a device, and the like installed at a work site, and may include a component, a module, and the like that constitute one equipment. Tasks include repair, maintenance, maintenance, adjustment, and operation. For example, a job could be maintenance on a problematic device, or troubleshooting a problematic device before the problem occurs.

The work site may be an industrial site such as a factory or a building, or it may be a home. For example, industrial sites can provide remote guidance for air conditioning, ventilation, air conditioning, fans, boilers, cooling towers, pumps, temperature and humidity sensors, refrigerators, lighting equipment, power equipment and fire systems in factories. Lighting, power, access control, disaster prevention, parking management, and facility management are possible through comprehensive management and monitoring of the entire building through the remote management device (2). , The worker can receive the remote guidance through the mobile terminal 3 and can smoothly perform work without the help of an expert.

According to another embodiment, the remote guidance system 1 may be operated on a home basis. For example, the remote management device 1 can be used to control household appliances such as home TVs, air conditioners and refrigerators, energy consumption devices such as water, electricity, and air conditioning, security devices such as door locks and surveillance cameras, And can be remotely adjusted while monitoring and controlling such devices. Also, when a problem occurs in the home appliance, if the customer has only the mobile terminal 3, remote guidance for problem solving can be provided, so that the problem of the appliance can be solved easily. In this case, the consumer does not need to contact the consumer center to call a specialist, or to pay the travel expenses separately, and also to avoid waste of time. It also eliminates the need for immersive displays that require traditional, separate hardware, such as smart glass.

The remote guidance system 1 can provide remote guidance to a plurality of IoT devices used in a smart device, a smart building, a smart factory, and the like. IoT devices can be digital signage, smart TV, smart watch, smart CE, drone, and so on. The remote guidance system 1 according to another embodiment may provide a working guidance to a legacy device manufactured before the IoT period in which there is no IoT sensing capability. For example, it is possible to provide a remote guidance using a portable terminal 3 for a legacy device which is not equipped with an IoT sensor built in or mounted in the device and can not sense information of the device and its surrounding environment.

The remote guidance system 1 according to an embodiment predicts an error for a plurality of devices and provides an error prediction based before service. A dictionary service is a service that predicts possible errors for devices and manages and processes errors before errors occur. For example, if an error of the device is predicted in advance through the remote management apparatus 2 and the device user is notified of the error, the user can easily receive an error by receiving the remote guidance using the portable terminal 3 possessed by the user So that it can be solved in advance.

For example, an operator located at a work site shoots a work site through an image acquisition unit such as a camera of the mobile terminal 3. [ Then, the virtual work guidance information automatically guiding the work order is combined with the photographed image on the screen of the portable terminal 3 and displayed.

The portable terminal 3 includes an image acquisition unit such as a camera, an output unit capable of outputting a screen, and a processor, which is an augmented reality engine for augmented reality-based image analysis. The portable terminal 3 acquires an actual image in a job site through an image acquisition unit, and displays an augmented reality content combining an acquired actual image with virtual job guidance information through an output unit. The virtual job guidance information displayed on the screen includes the identifier information, the position information, the status information, the job order information, the job tool information, and the like for the job subject. The identifier information is a virtual object for identifying the work object. For example, it informs the worker of an object such as a dotted line or a block. The position information is a virtual object for informing the position of the work object. For example, a virtual finger or the like informs the position of the equipment or the parts in the equipment that require work. The work order information is a virtual object for informing guide information according to a sequential work order. The work tool information is a virtual object for informing the tool necessary for the work. For example, it displays all the virtual tools needed for a task, such as a driver, a hammer, and so on. An embodiment of the job guidance information displayed on the screen will be described later with reference to Figs. 2, 3 and 9. Fig.

In order to apply the augmented reality to the actual image photographed in real time, the portable terminal 3 according to an embodiment repeats the sequential processes of object recognition, object tracking, and object rendering to link the virtual objects to the actual image. For example, the mobile terminal 3 recognizes an actual object in an actual image, tracks the recognized actual object, combines and renders a virtual object corresponding to the augmented reality-based operation guidance information on the actual object being tracked, A series of processes of rendering a virtual object and combining it with an actual object to generate an augmented reality content is repeated. Since the actual image captured through the camera is not fixed but moves continuously due to the motion of the operator, it is necessary to follow the motion of the recognized object in the actual image, which is called "object tracking". "Object Rendering" is a process of creating a three-dimensional image by injecting realism into a two-dimensional image in consideration of external information such as light source, position, and color. A wire frame, and a ray tracing rendering method. Rendering may be performed in the portable terminal 3, but in a distributed environment, the rendering may be performed in the remote management device, and the result of the rendering may be transmitted to the portable terminal 3.

The remote guidance system 1 according to an embodiment uses augmented reality-based image analysis and machine learning together when recognizing an object. The state of the equipment to be recognized in the actual image may be various, such as steady state, some rusted state, and distorted state. Therefore, the recognition accuracy can be lowered only by the engine used in the augmented reality. Therefore, it improves object recognition rate by complementing it with machine learning. Machine learning based object recognition can be performed in the remote management apparatus 2, and augmented reality based object recognition can be performed in the portable terminal 3. Or both in the remote management apparatus 2 and the portable terminal 3 according to the embodiment. In the distributed environment, the remote management apparatus 2 can have a negotiation process for improving the accuracy by comparing the state of the equipment recognized through the machine learning with the state of the equipment recognized by the mobile terminal 3. An embodiment of the negotiation process will be described below with reference to FIG.

The remote guidance system 1 according to an exemplary embodiment provides a scripting tool for easily creating augmented reality-based job guidance information for a user who wants to produce the augmented reality-based operation guidance information. Therefore, it can be manufactured without the cost of additional augmented reality manual production. When providing a scripting tool, you can provide a number of templates that can be applied samples by field. Manufacturers can be small-scale operators and equipment suppliers. For example, an author can easily create augmented reality-based job guidance information using templates provided with a scripting tool with a user terminal, for example, a desktop. The portable terminal 3 can receive the augmented reality based job guidance information produced in this way and use it for creating the augmented reality contents.

The portable terminal 3 includes all portable user terminals such as a smart phone and a tablet PC carried by the user. The worker can receive work guidance through the portable terminal 3 without the need for an immersive display requiring an existing separate hardware such as a smart glass. Of course, when the operator wears an immersive display, it can work in conjunction with the display.

The remote management device (2) monitors equipment located at the work site and performs remote engineering on the equipment that needs work. At this time, if an error occurs in a predetermined device, the remote management apparatus 2 can perform remote engineering to solve the error. As another example, each piece of equipment at the shop floor can be monitored to predict the error in advance and remote engineering can be performed to handle errors for the equipment before the error occurs. Examples of remote engineering include video calls, chat, remote screen sharing, and remote control. When the worker can not solve the problem through the augmented reality-based job guidance information provided through the mobile terminal 3, the portable terminal 3 is connected to the remote management apparatus 2 to receive remote control from the expert have. The remote management apparatus 2 may be in the form of a server and may be designed based on a cloud platform in consideration of scalability.

2 is a configuration diagram of a remote guidance system for explaining the concept of remote guidance according to an embodiment of the present invention.

Referring to FIG. 2, the remote guidance system recognizes an object to be operated in an actual image by using the augmented reality-based image analysis of the mobile terminal 3 and the machine learning of the remote management apparatus 2. To this end, the portable terminal 3 shares the photographed actual image with the remote management apparatus 2. [ The remote management apparatus 2 recognizes an object in the actual image using the learning data accumulated through machine learning in advance. The machine learning based object recognition result is transmitted to the portable terminal 3, and the recognized object identifier (ID) can be included. For example, the remote management apparatus 2 transmits to the portable terminal 3 an ID1: power half cover, ID2: breaker, and ID3: power supply, which are object identifiers recognized through machine learning in an actual image. The mobile terminal 3 recognizes the object in the actual image through the augmented reality-based image analysis and raises the accuracy of object recognition by reflecting the machine learning based object recognition result of the remote management apparatus 2 to the augmented reality based recognition result .

The portable terminal 3 displays the augmented reality contents on the screen by combining the operation guidance information with the actual images including the recognized objects. For example, as shown in FIG. 2, if there is a problem in the power supply of the power supply in the power supply, the display of the portable terminal displays a warning " Power must be reduced because the power of the power supply exceeds the threshold value " Status information 102 such as a message is displayed. Then, identification information 100 such as an identifier such as a block type is displayed on the screen on the power half cover in which the problem occurs, so that the operator can recognize that the power half cover is on the work bench. The identification information 100 can be changed in accordance with the operation order.

The work order 104 is also displayed on the screen. For example, a pre-set operation sequence is displayed on the screen according to the operation manual, such as "1. Open the power panel cover 2. Lower the breaker 3. Replace the power supply 4. Raise the breaker". In the "1. Open the power panel cover", the handle of the power panel cover in the actual image can be identified by dotted lines. In "2. Breaking down the breaker", the breaker in the actual image can be displayed discretely by a dotted line or the like. In "3. Replace power supply", the power supply in the actual image can be indicated by dotted lines.

The job guidance information described above can be prepared in advance through the scripting tool 106 dynamically editable according to the job manual based on the predefined template. If the worker can not solve the problem using the augmented reality-based job guidance information, the portable terminal 3 can be connected to the remote management apparatus 2 via a network and receive remote engineering support from the expert through the remote management 108 . Remote engineering includes video call, chat, remote screen sharing, remote control, and so on.

3 is a block diagram of a remote guidance system for explaining the concept of remote guidance according to another embodiment of the present invention.

Referring to FIG. 3, when the object of operation is an engine, the remote management apparatus 2 transmits a machine learning based object recognition result to the portable terminal 3. For example, the remote management apparatus 2 transmits ID1: radiator fan, ID2: timing belt, and ID3: spark plug, which are object identifiers recognized through machine learning in the actual image, to the portable terminal 3. The mobile terminal 3 recognizes the object in the actual image through the augmented reality-based image analysis and raises the accuracy of object recognition by reflecting the machine learning based object recognition result of the remote management apparatus 2 to the augmented reality based recognition result .

The portable terminal 3 displays the augmented reality contents on the screen by combining the operation guidance information with the actual images including the recognized objects. For example, as shown in FIG. 3, when a problem occurs in the engine, identification information 100 such as an identifier such as a dotted line in the form of a dotted line is displayed on a screen for an engine so that the operator can recognize . The identification information 100 can be changed in accordance with the operation order.

A work order 104 is displayed on the screen. For example: 1. Open the bonnet 2. Check the timing belt tension 3. Confirm the radiator fan rotation 4. Confirm the spark plug 4. Depending on the operation manual, . After the operator opens the bonnet, the timing belt in the actual image can be displayed in a distinguishable manner with a dotted line or the like. After the worker confirms the tension of the timing belt, the radiator in the actual image can be displayed discretely by a dotted line or the like. After the worker confirms the rotation of the radiator fan, the actual in-image spark plug can be discernibly displayed with a dotted line or the like.

The job guidance information described above can be prepared in advance through the scripting tool 106 dynamically editable according to the job manual based on the predefined template. For example, a user can use the scripting tool 106 to create job guidance information in accordance with a job manual through a user terminal 4 such as a desktop. The scripting tool 106 enables a user to easily create augmented reality-based job guidance information without a manufacturing cost of an additional augmented reality manual. When providing the scripting tool 106, it is possible to provide a plurality of templates that can be applied samples by field. Users can be small business and equipment suppliers. For example, a user can easily create augmented reality-based job guidance information using a template provided with the scripting tool with the user terminal 4. [ The portable terminal 3 can receive the augmented reality based job guidance information produced in this way and use it for creating the augmented reality contents. The template drawing 40 in which the virtual control area is schematically displayed may be displayed on the control screen. A graphic object representing the virtual tool may be placed in the template drawing.

If the worker can not solve the problem using the augmented reality-based job guidance information, remote engineering can be supported through a remote management 108 from a specialist via a network connection with the remote management apparatus 2. [ Remote engineering includes video call, chat, remote screen sharing, remote control, and so on.

4 is a configuration diagram of a portable terminal according to an embodiment of the present invention.

4, the portable terminal 3 includes an image acquisition unit 30, an input unit 31, a processor 32, an output unit 33, a communication unit 34, and a memory 35.

The image acquiring unit 30 acquires an actual image photographed at the current time of the operator in real time and transmits the acquired real image to the processor 32. [ The image acquiring unit 30 may be, for example, a camera, an image sensor, or the like. At this time, a sensor for detecting the movement of the user or the motion of the portable terminal 3 may be additionally provided.

The input unit 31 receives a signal input by a key input of the user and transmits the signal to the processor 32. The key includes, for example, a photographing button for photographing the actual image, a menu key for controlling the operation guidance information, and a power button for powering the portable terminal 3.

The output unit 33 outputs the processed information through the processor 32. [ The output unit 33 according to an embodiment outputs the augmented reality contents generated through the processor 32. [ An example of the output is an image display on the screen, and a voice output through a speaker.

The communication unit 34 is connected to the remote management apparatus via a network, and performs wired / wireless communication. The communication unit 34 interacts with the remote management apparatus to maintain the homeostasis of the network. The portable terminal 3 may be connected to the remote management apparatus via a dedicated network through the communication unit 34 or may be connected through a general-purpose network such as the Internet. The communication unit 34 according to the embodiment starts the job guidance upon receiving the job command from the remote management apparatus. For example, when a remote management apparatus monitors equipment in a worksite and detects an error or an equipment that may cause an error, the remote management apparatus transmits a work request command for the equipment to the portable terminal to inform the operator.

The communication unit 34 according to the embodiment transmits the actual image acquired through the image acquisition unit 30 to the remote management apparatus and shares the same. Then, the recognized object information is transmitted to the remote management apparatus through the augmented reality-based image analysis of the mobile terminal, and the recognized object information can be received from the remote management apparatus through the machine learning.

The memory 35 stores data. Data may be unique information of the equipment at the worksite, location information, drawings of the equipment, job guidance information, part information in the equipment, job schedule information, and the like. Such information may be downloaded from the remote management apparatus and received from the user through the input unit 31. [ The memory 35 may be a CF (Compact Flash Card), an SD card (Secure Digital Card), an SM card (Smart Media Card), an MMC Such as a memory card, a multi-media card, or a memory stick, or may be provided in a separate apparatus.

The processor 32 controls the operation of each of the above-described configurations. The processor 32 according to an embodiment repeatedly performs object recognition, tracking, and rendering on a real image acquired through the image acquisition unit 30. [ For example, it is possible to recognize a work object in an actual image, to match a virtual work guidance information to assist a work on a work target while tracking a recognized work target, and to render the virtual work guidance information by rendering the virtual work guidance information Then, the augmented reality contents are generated by matching with the actual images. The processor 32 in accordance with one embodiment processes commands for remote control of the remote management device. For example, the portable terminal screen is shared with the remote management apparatus.

The processor 32 in accordance with one embodiment uses augmented reality-based image analysis and learning data-based machine learning for object recognition. Image analysis can use a general algorithm to recognize objects in the acquired real image. For example, objects are recognized using point cloud information, depth information, color information, and the like in an image. Since object recognition based on image analysis is a known technique, a detailed description will be omitted. At this time, it is possible to increase the object recognition rate by processing image processing, for example, binarization, edge blending, and masking for the image.

Machine learning is one of the methods used in artificial intelligence. It is a technology that learns by inputting a lot of data into a computer and classifying similar things. For example, if a photo similar to a saved dog photo is entered, the computer will classify it as an open photo. The object in the image can be recognized by using the learning data accumulated through the machine learning. The processor 32 uses the object recognition using the machine learning as an auxiliary means of the augmented reality-based image analysis to enhance the accuracy of object recognition. Machine learning can use Support Vector Machines (SVM) and Deep Learning techniques.

When the portable terminal 3 is connected to the remote management apparatus, the portable terminal 3 becomes an agent that executes a control command received from the remote management apparatus. The mobile terminal 3 can execute the application for remote guidance. At this time, it has a form capable of supporting various OSs such as Android, iOS, and Windows, which are different from each other.

FIG. 5 is a detailed configuration diagram of the processor of FIG. 4 according to an embodiment of the present invention.

4 and 5, the processor 32 includes an image processing unit 320, a remote control processing unit 324, and a device control unit 326. [ The image processing unit 320 includes an object recognition unit 3200, a tracking unit 3202, an augmented reality processing unit 3204, and a visualization unit 3206.

The object recognition unit 3200 recognizes an object in an actual image using an augmented reality-based image analysis on the real image and a machine learning using the learning data. The object recognition unit 3200 according to an embodiment recognizes an object by reflecting the result of recognition of a machine learning based object of the remote management apparatus received through the communication unit 34 on the augmented reality based object recognition result.

The tracking unit 3202 tracks the movement of the object recognized by the object recognition unit 3200. Since the actual image captured through the camera is not fixed but moves continuously due to the motion of the operator, the tracking unit 3202 performs the operation of following the motion of the recognized object in the actual image.

The augmented reality processing unit 3204 calculates the interaction between the real environment and the virtual environment and generates the augmented reality contents by reflecting the calculation result. The augmented reality content is a virtual environment combined with a real environment. Accordingly, the operation guidance information can be provided based on the augmented reality. The augmented reality processing unit 3204 according to an exemplary embodiment calculates an interaction between a real object and a virtual object, reflects the calculation result, combines the virtual object with the actual image and then renders or renders the virtual object, Thereby generating augmented reality contents. At this time, the virtual object position to express the work guidance information in the real image can be selected and the work guidance information can be arranged according to the work order at the selected virtual object position. The augmented reality processing unit 3204 according to an exemplary embodiment receives job guidance information generated in advance through a scripting tool capable of dynamically processing according to a job manual based on a predefined template, and uses the job guidance information to generate augmented reality content . The visualization unit 3206 provides the output unit 33 with the augmented reality contents generated through the augmented reality processing unit 3204.

The remote control processing unit 324 requests the remote management apparatus for remote control and remotely shares the screen according to the control of the remote management apparatus. The device control unit 326 processes the control request of the remote management apparatus.

6 is a detailed configuration diagram of the output unit of FIG. 4 according to an embodiment of the present invention.

6, the output unit 33 includes an identifier display unit 330, a status display unit 332, a work order display unit 334, and a work tool display unit 336. [

The identifier display unit 330 displays a virtual object for identifying a work object on the screen. For example, an identifier such as an arrow, a line, or a block is displayed on the screen. At this time, the work subject can be changed according to the work order. Actual images shot in real-time continue to move, so identifiers also move to the target of moving objects. The identifier display unit 330 may notify the position of the work target. For example, it refers to a part requiring work by hand or the like.

The status display unit 332 displays various status information about the equipment such as the control status of the operation target, the failure status, the operation status, the time information, and the control parameters required for operation in the equipment. At this time, a detailed description including the part number of the work subject, the life cycle, the recent replacement history, the failure history, and the like may be displayed on the screen. For example, when the equipment is an air conditioner, the control parameter may be a target temperature (or set temperature), an upper limit temperature, a lower limit temperature, an air flow rate, an operation stoppage, and the like. If the equipment is a lighting fixture, the control parameters may be illumination brightness, and so on. The status display unit 332 can automatically display status information for equipment that requires work, but may also display status information at the request of an operator. For example, when an operator selects a predetermined equipment in the screen, various status information about the selected equipment can be displayed on the screen.

The work order display unit 334 displays a virtual object for informing a sequential work order on the screen. At this time, it is possible to notify the worker through the identifier display part 330 of the equipment or the parts in the equipment that need work in accordance with the work order. The work tool display unit 336 displays a virtual object for informing a tool necessary for the work on the screen. Work tools include, for example, a hammer, a screwdriver, a hand, and the like.

7 is a configuration diagram of a remote management apparatus according to an embodiment of the present invention.

7, the remote management apparatus 2 includes a verification management module (VMM) 20, a monitoring screen module (MSM) 21, a network handling module (NHM) 22 A Remote Control & Control Module (RCM) 23, a Data Processing Module (DPM) 24, a Device & App Monitoring Module (DMM) 25, Stack (MPS) 26 and Bigdata Statistics Engine (BSE) 27. The remote management apparatus 2 can be designed on the basis of a cloud platform in consideration of expandability.

The authentication management unit 20 verifies the validity of the user information of the requested worker and authenticates who is the subject of the request received by the remote management apparatus 2 as necessary. The monitoring unit 21 monitors the screen of the portable terminal of the remote worker. At this time, it provides the function to monitor or manipulate through the terminal of the technician and the manager.

The network connection unit 22 maintains the consistency of the network connection irrespective of the type of the network. The network connection unit 22 transmits / receives information to / from the portable terminal of the operator. For example, the network connection unit 22 receives and saves the actual image from the portable terminal, receives the recognized object information through the image analysis based on the augmented reality of the portable terminal, and outputs the machine learning based object recognition result to the portable terminal send. The remote control unit 23, when receiving a remote control request from the portable terminal, shares the screen of the portable terminal and transmits a remote control command to the portable terminal. At this time, it provides necessary process adjustment function.

The data processing unit 24 recognizes the object through machine learning based on the learning data on the actual image received from the portable terminal through the network connection unit 22. [ The client management unit 25 confirms the status of the client, and if the client is operating normally or not, the client management unit 25 detects and identifies a problem. The client may be a mobile terminal carried by an operator, an application executed in a mobile terminal, or various kinds of equipment in a work site. The mobile terminal combines the augment information based on the augmented reality with the actual image taken by the operator. The client management unit 25 according to an embodiment monitors the equipment in the workplace to predict an error in advance and generates a work command for processing an error for the corresponding equipment before an error occurs. The generated work command is transmitted to the portable terminal of the worker through the network connection unit (22).

The protocol processing unit 26 is a software module that implements the communication protocol for receiving and parsing the normal signal of the client and processing the signal. The data analysis unit 27 analyzes the data received from the client so as to obtain a stable service by grasping an abnormal symptom of a plurality of clients. At this time, the analyzed data can be compared with a preset threshold value, and an abnormal symptom can be grasped.

8 is a flowchart illustrating an object recognition process through cooperation between a remote management apparatus and a mobile terminal according to an embodiment of the present invention.

Referring to FIG. 8, the portable terminal 3 first shares the photographed actual image with the remote management apparatus 2 (700), and first recognizes the actual image object through the augmented reality-based image analysis (710) . Based on the collaboration with the remote management apparatus 2, the object is secondarily recognized by reflecting the object recognition result through the machine learning in the remote management apparatus to the object recognition result through the augmented reality based image analysis.

Collaboration between the remote management apparatus 2 and the mobile terminal 3 is performed by comparing the state of the equipment recognized by the machine learning with the state of the equipment recognized by the mobile terminal 3 by the remote management apparatus 2 Height is the process. Hereinafter, a negotiation process between the remote management apparatus 2 and the mobile terminal 3 will be described with reference to FIG.

First, when the portable terminal 3 captures an actual image through the camera 70, the captured real image is shared with the remote management apparatus 2 (700). At this time, the portable terminal 3 recognizes the object in the actual image through the augmented reality-based image analysis in the image processing unit 710. The remote management apparatus 2 recognizes the object in the actual image using the learning data learned through the machine learning in the data processing unit (operation 720). The portable terminal 3 continuously tracks the recognized object together with the object recognition 710 (740). Then, the virtual job guidance information is rendered 750 and combined with the actual environment including the tracked object, or the virtual job guidance information is combined with the actual environment including the tracked object and then rendered 750 to generate the augmented content And outputs it to the screen. Object recognition, tracking, and rendering operations can be performed iteratively.

Hereinafter, the object recognition process through cooperation with the remote management apparatus 2 will be described. The portable terminal 3 transmits the recognized object information to the remote management apparatus 2 (730). For example, it inquires whether the object recognition is correct while transmitting the recognized object identifier ID: 002. The remote management apparatus 2 having received the object information recognized from the portable terminal 3 transmits the recognized object information to the portable terminal 3 through machine learning (732). For example, it transmits ID: 001, which is an object identifier recognized through machine learning, to the portable terminal 3. Since the mobile terminal 3 having received the object identifier ID 001 is different from the recognized object identifier, the remote terminal 2 requests the remote management apparatus 2 to perform object recognition again (734). Then, the remote management apparatus 2 performs the machine learning based object recognition once again, and transmits the recognized object identifier ID: 001 to the portable terminal 3 (736). The mobile terminal 3 having received the object identifier ID: 001 changes the recognized object identifier to ID: 001 and transmits an OK response message to the remote management apparatus 2 (738).

FIG. 9 is a portable terminal screen for illustrating an example in which job guidance information is displayed on a portable terminal according to an embodiment of the present invention.

Referring to FIG. 9, an identifier 900 for identifying a work object is displayed on the portable terminal screen. At this time, it is possible to further display an identifier 910 for identifying a part requiring work in the work subject. At this time, on the portable terminal screen, status information indicating the state of the work subject, a sequential work order 904, a work tool 906 necessary for the work, and the like can be displayed together.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

1: Remote guidance system 2: Remote management device
3: portable terminal 20: authentication management unit
21: Monitoring section 22: Network connection section
23: remote control unit 24: data processing unit
25: client management unit 26: protocol processing unit
27: Data analysis unit 30: Image acquisition unit
31: input unit 32: processor
33: output unit 34: communication unit
35: memory 320:
324: remote control processing unit 326:
330: identifier display section 332: status display section
334: Operation order display section 336: Operation tool display section
3200: Object recognition unit 3202:
3204: augmented reality processing unit 3206:

Claims (14)

An image acquiring unit for acquiring an actual image photographed at a current time of the operator;
A processor for recognizing a work object on the acquired actual image and generating virtual reality content by matching virtual work guidance information for assisting work on the work object while tracking the recognized work object to the actual image including the tracked work object, ; And
An output unit for outputting the generated augmented reality contents;
And a portable terminal. 2. The system of claim 1, wherein the processor
An object recognition unit for recognizing an object in an actual image using augmented reality - based image analysis and machine learning;
A tracking unit for tracking movement of the recognized object;
An augmented reality processing unit for calculating an interaction between a real environment including a tracked object and a virtual environment and reflecting the calculation result to generate augmented reality contents combining a virtual environment with a real environment to provide work guidance information based on the augmented reality; And
A visualization unit for providing the output unit with the augmented reality contents generated through the augmented reality processing unit;
And a portable terminal. The mobile terminal of claim 2,
The remote management apparatus communicates with the remote management apparatus via a network, and the actual image obtained through the image acquisition unit is shared with the remote management apparatus, and the recognized object information is transmitted to the remote management apparatus through the augmented reality based image analysis of the mobile terminal. A communication unit for receiving object information recognized through machine learning; Further comprising:
Wherein the object recognition unit recognizes the object by reflecting the machine learning based object recognition result of the remote management apparatus received through the communication unit to the augmented reality based object recognition result. 3. The apparatus of claim 2, wherein the augmented reality processing unit
The virtual reality object is generated by combining the virtual object with the actual image after rendering the actual object and the virtual object, calculating the interaction between the actual object and the virtual object, combining the virtual object with the actual image, rendering the virtual object, Selecting a virtual object location to express information, and arranging work guidance information in accordance with a work order at a selected virtual object location. 3. The apparatus of claim 2, wherein the augmented reality processing unit
Wherein the mobile terminal receives input of work guidance information generated in advance through a scripting tool that can be dynamically processed according to a job manual based on a predefined template and uses the job guidance information to generate an augmented reality content. 3. The system of claim 2, wherein the processor
A remote control processor for requesting a remote control device for remote control and sharing the screen of the portable terminal remotely according to the remote control of the remote control device; And
A device control unit for processing a control request of the remote management apparatus;
Further comprising: 2. The apparatus of claim 1, wherein the output
An identifier display unit for displaying a virtual object for identifying a work object on a screen;
A status display unit for displaying a virtual object informing a status of a work subject on a screen;
A work order display unit for displaying a virtual object guiding a sequential work order on a screen; And
A work tool display unit for displaying on the screen a virtual object informing a tool necessary for the work;
And a portable terminal. A client management unit for managing a mobile terminal that combines augmented reality based job guidance information with an actual image photographed by a worker at a job site;
A network connection unit for receiving and sharing an actual image from a portable terminal, receiving recognized object information through image analysis based on an augmented reality of the portable terminal, and transmitting the machine learning based object recognition result to the portable terminal; And
A data processor for recognizing the object using the machine learning based learning data on the received actual image;
The remote management apparatus comprising: 9. The method of claim 8,
The client management unit monitors equipment in a worksite to predict an error in advance and generates a work command for processing an error for the equipment before an error occurs,
And the network connection unit transmits the created work command to the mobile terminal. 9. The system according to claim 8, wherein the remote management apparatus
A remote controller for receiving a remote control request from a portable terminal, remotely sharing a screen of the portable terminal, and transmitting a remote control command to the portable terminal to perform remote engineering;
Wherein the remote guidance system further comprises: 9. The system according to claim 8, wherein the remote management apparatus
Wherein the remote management device operates based on the cloud. A remote guidance method using a mobile terminal, the mobile terminal comprising:
Obtaining an actual image photographed at a current time of the operator;
Recognizing a work object from the obtained actual image;
Tracking the recognized work object;
Generating augmented reality contents by matching virtual work guidance information for assisting an operation on a work subject to an actual image including a tracked work object; And
Outputting the generated augmented reality contents;
The remote guidance method comprising: 13. The method of claim 12, wherein recognizing the object comprises:
Sharing a photographed actual image with a network-connected remote management apparatus;
Recognizing an object in a real image through augmented reality-based image analysis;
A second recognition step of recognizing an object based on the collaboration with the remote management apparatus, the object recognition result through the augmented reality based image analysis by reflecting the object recognition result through the machine learning in the remote management apparatus;
The remote guidance method comprising: 14. The method of claim 13, wherein the secondary recognition of the object
Transmitting the recognized object information to the remote management apparatus through the augmented reality-based image analysis;
Receiving object information recognized through machine learning from a remote management apparatus;
Comparing the object information received from the remote management apparatus with the object information recognized through the mobile terminal, and requesting the remote management apparatus to perform object recognition again if the object information is different;
Receiving a machine learning based object recognition execution result from a remote management apparatus;
Modifying the object recognition result of the mobile terminal by reflecting the received object recognition result;
The remote guidance method comprising:

Images