KR102088012B1 - System and method for maintaining based on augmented reality using wearable oscilloscope - Google Patents

Abstract

The present invention relates to an augmented reality based maintenance system using a wearable oscilloscope comprises: a head mounted display device identifying equipment subject to maintenance including a plurality of components, and providing a maintenance worker with maintenance measure information based on the state of the equipment subject to maintenance; and a wearable device measuring the state of the equipment subject to maintenance and generating measurement result information. The head mounted display device comprises: an image photographing unit photographing at least one area of the equipment subject to maintenance and generating an input image; an object recognizing unit acquiring a spectrum pattern connected to each of the plurality of components based on drawing data by component of the equipment subject to maintenance, and identifying the components of the equipment subject to maintenance included in the input image based on the spectrum pattern; and an output display unit displaying an output image in which at least one between technical manual information of a relevant component and the measurement result information overlaps the input image based on the identification results. In addition, the wearable device comprises: a measuring unit including the oscilloscope for measuring an electric signal of the component and generating the measurement result information; a communication module transmitting the measurement result information to the head mounted display device; a control unit including GPU for controlling graphic processing of the output display unit; and a power supply unit supplying power for operating the measuring unit. Therefore, the augmented reality based maintenance system using a wearable oscilloscope can be utilized even inside a small space and can allow even an unskilled person to conduct work such as repair and maintenance of equipment subject to maintenance.

Description

Augmented reality-based maintenance method and system using wearable oscilloscope {SYSTEM AND METHOD FOR MAINTAINING BASED ON AUGMENTED REALITY USING WEARABLE OSCILLOSCOPE}

The present application relates to augmented reality-based maintenance method and system using a wearable oscilloscope.

In general, an oscilloscope is a device that can measure the voltage change of a specific time interval or a specific band. Oscilloscopes can identify voltage signals that change over time in the form of periodic and repetitive waveforms.

On the other hand, most oscilloscope equipment is composed of a measuring probe and the main body in contact with the target to measure the electrical signal, such as voltage, the conventional oscilloscope equipment belonged to a large and heavy equipment was difficult to use in a narrow work space However, there was a restriction that the user must carry the body when moving from the work space.

Particularly in military equipment, for example, due to the narrow space of the control room such as Oricon, Cheongung, and Cheonma, which are anti-aircraft weapons, the mechanic measures, repairs, or repairs the electrical signals of the electronic equipment in the anti-aircraft weapons using conventional oscilloscope equipment. It is difficult to carry out such tasks, and in the case of military equipment, it should be able to operate immediately even in case of sudden battles. The technical know-how is not passed on, and tends to be dead, so it is necessary to shorten the time required to learn maintenance manuals and acquire practical practical experiences, and for the non-skilled person to easily perform simple measurement, repair, and maintenance. The need for technology development is growing.

Background art of the present application is disclosed in Korea Patent Publication No. 10-1701278.

The present invention is to solve the above-mentioned problems of the prior art, by using an oscilloscope-mounted wearable device can measure the electrical signal of the equipment to be repaired, maintenance manual information associated with the parts of the equipment to be repaired and the electrical signal By outputting the measurement results to the head mounted display (HMD) device that can be worn by the operator based on augmented reality, it can be utilized in a small space and the unskilled person can smoothly perform the repair and maintenance of the equipment to be maintained. It is an object of the present invention to provide augmented reality-based maintenance system and method using a wearable oscilloscope.

The present application is to solve the above-mentioned problems of the prior art, using a three-dimensional object recognition algorithm to identify the parts of the equipment to be maintained with high accuracy, maintenance manual including the maintenance method of the corresponding parts in connection with the identification results By displaying the information on the basis of augmented reality, the operator can easily identify the part and acquire the know-how required to maintain the part, and in conjunction with the wearable oscilloscope, it enhances the result of measuring the electrical signal of the part of the equipment under maintenance. An object of the present invention is to provide an augmented reality based maintenance system and method using a wearable oscilloscope that can be displayed based on reality.

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, augmented reality-based maintenance system using a wearable oscilloscope according to an embodiment of the present application, to identify the maintenance target equipment including a plurality of parts, and to the state of the maintenance target equipment And a head mounted display device configured to provide maintenance action information to a mechanic based on the condition, and a wearable device configured to measure a state of the maintenance target equipment to generate measurement result information.

The head mounted display apparatus may further include an image photographing unit configured to photograph at least one region of the maintenance target equipment to generate an input image, and to link the plurality of components to each of the plurality of components based on the plurality of component drawing data of the maintenance target equipment. At least one of maintenance manual information of the corresponding part and the measurement result information based on the identification result and an object recognizing unit for acquiring the acquired spectral pattern and identifying the part of the maintenance target equipment included in the input image based on the spectral pattern; It may include an output display unit for displaying an output image overlapping one of the input image.

The wearable device may further include a measurement unit having an oscilloscope for measuring an electrical signal of the component to generate the measurement result information, a communication module for transmitting the measurement result information to the head mounted display device, and graphic processing of the output display unit. It may include a control unit having a GPU for controlling the power supply unit for supplying power for driving the measurement unit.

In addition, the object recognition unit, the point of receiving the drawing data of the plurality of parts of the maintenance target equipment including the plurality of parts, and designating the point specifying the shape of the part on the drawing data to perform a point index And a data logging unit for deriving spectral information for each point by repeating a three-dimensional rotation to the modeling result and a machine learning unit for patterning the spectral information by performing machine learning on the spectral information.

In addition, the head mounted display device may include an AR glass.

The image capturing unit may generate the input image by a camera provided in the AR glass.

The output display unit may amplify and output the output image on the display of the AR glass.

The head mounted display apparatus may further include an image corrector configured to determine the resolution of the output image based on the AR glass standard and to determine the sharpness of the output image based on the illuminance of an external environment.

The component may include a circuit card, and the measuring unit measures an electrical signal of the circuit card to detect an abnormal waveform by comparing with a preset reference value, and when the abnormal waveform is detected, the circuit. Information indicating that the card needs to be replaced may be included in the measurement result information.

In addition, the head mounted display device may be formed in the form of a helmet or band that can be worn on the head of the mechanic.

In addition, the wearable device may be formed in the form of a vest or a jacket that can be worn on the body of the mechanic.

In addition, the maintenance target equipment may be an altitude detection radar of an anti-aircraft guided weapon.

The maintenance manual information may include at least one of repair accessory specification information, assembly procedure information, disassembly procedure information, and inspection reference information of the corresponding component.

On the other hand, augmented reality-based maintenance method using a wearable oscilloscope according to an embodiment of the present application, receiving an input image photographing at least one region of the maintenance target equipment including a plurality of parts, the input based on the spectral pattern Identifying parts of the maintenance target equipment included in the image, measuring electrical signals of the parts through an oscilloscope to generate measurement result information, and at least one of maintenance manual information and the measurement result information of the identified parts And displaying an output image overlapping the input image.

The identifying of the parts of the maintenance target equipment may include receiving drawing data of the plurality of parts of the maintenance target equipment including the plurality of parts, and specifying a shape of the parts on the drawing data. Performing modeling by specifying a symbol, deriving spectral information for each point by repeating a three-dimensional rotation in the modeling result, and performing machine learning on the spectral information to pattern the spectrum information to form a spectrum pattern. And acquiring parts of the maintenance target equipment included in the input image based on the obtaining and the spectral patterns.

The receiving of the input image may include receiving the input image photographed by a camera provided in a head mounted display apparatus including an AR glass.

The displaying of the output image may include augmenting and outputting the output image on the display of the head mounted display apparatus including the AR glasses.

In addition, the AR-based maintenance method using a wearable oscilloscope according to an embodiment of the present invention, the step of determining the resolution of the output image based on the standard of the AR glass and the sharpness of the output image based on the illumination of the external environment Determining may be further included.

The component may include a circuit card, and the generating of the measurement result information may include detecting an abnormal waveform by measuring an electrical signal of the circuit card and comparing it with a preset reference value and the abnormal waveform. If this is detected, it may include the step of including information in the measurement result information that the replacement of the circuit card is necessary.

The maintenance manual information may include at least one of repair accessory specification information, assembly procedure information, disassembly procedure information, and inspection reference information of the corresponding component.

The above-mentioned means for solving the problems are merely exemplary, and should not be construed as limiting the present application. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, it is possible to measure the electrical signal of the equipment to be repaired using the wearable device mounted with an oscilloscope, and to maintain the maintenance manual information and the measurement result of the electrical signal associated with the parts of the equipment to be repaired. It is effective to provide an augmented reality based maintenance system and method using a wearable oscilloscope that can output a self-wearing head mount display (HMD) device based on augmented reality.

According to the above-described problem solving means of the present application, the oscilloscope for measuring the electrical signal is provided on the wearable device that can be worn by the mechanic, and the maintenance target equipment on the head-mounted display device that can be worn by the mechanic without having to carry a separate maintenance manual. By displaying the information associated with the repair or maintenance of the equipment, even in the case of a non-skilled person in a narrow space, the electrical signal measurement, repair work, maintenance work, etc. of the equipment to be maintained can be performed smoothly.

According to the aforementioned problem solving means of the present application, it is possible to identify the parts of the maintenance target equipment with high accuracy by using a three-dimensional object recognition algorithm.

According to the above-described problem solving means of the present application, it is possible to shorten the time required for the mechanic to acquire the information necessary for the maintenance of the maintenance target equipment and acquire the know-how.

However, the effects obtainable herein are not limited to the effects as described above, and other effects may exist.

1 is a schematic diagram of augmented reality-based maintenance system using a wearable oscilloscope according to an embodiment of the present application.
2 is a schematic diagram of a head mounted display device according to an exemplary embodiment of the present disclosure.
3 is a schematic diagram of an object recognition unit according to an exemplary embodiment of the present application.
Figure 4 is a schematic diagram of a wearable device according to an embodiment of the present application.
5 is an operation flowchart of an augmented reality-based maintenance method using a wearable oscilloscope according to an embodiment of the present application.
6 is a flowchart of a method of identifying a component of a serviceable device according to an embodiment of the present application.
7 is a flowchart illustrating a method of generating measurement result information by measuring an electrical signal of a component of a maintenance target device according to an exemplary embodiment of the present application.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present disclosure. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted for simplicity of explanation, and like reference numerals designate like parts throughout the specification.

Throughout this specification, when a part is "connected" to another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element in between. "Includes the case.

Throughout this specification, when a member is said to be located on another member "on", "upper", "top", "bottom", "bottom", "bottom", this means that any member This includes not only the contact but also the case where another member exists between the two members.

Throughout this specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding the other components unless otherwise stated.

1 is a schematic diagram of augmented reality-based maintenance system using a wearable oscilloscope according to an embodiment of the present application.

Referring to FIG. 1, an augmented reality based maintenance system 10 using a wearable oscilloscope according to an exemplary embodiment of the present disclosure may include a head mounted display device 100, a wearable device 200, and a maintenance target device 300. .

The head mounted display apparatus 100 may identify the maintenance target equipment 300 including a plurality of parts, and provide maintenance personnel with maintenance measures information based on the state of the maintenance target equipment 300.

According to the exemplary embodiment of the present application, the maintenance measure information may include maintenance manual information corresponding to a part of the maintenance target equipment 300 to be described later and measurement result information generated by the measurement unit 210.

In addition, according to the exemplary embodiment of the present disclosure, the head mounted display apparatus 100 may include an AR glass.

In addition, according to the exemplary embodiment of the present disclosure, the head mounted display apparatus 100 may be formed in the form of a helmet or a band that may be worn on the head of the mechanic.

The wearable device 200 may measure the state of the maintenance target device 300 to generate measurement result information.

In addition, according to one embodiment of the present application, the wearable device 200 may be formed in the form of a vest or a jacket that can be worn on the body of the mechanic.

According to one embodiment of the present application, the maintenance target equipment 300 may be military equipment such as military vehicles, tanks, armored vehicles, missile launch vehicles, radars, and the like.

According to one embodiment of the present application, the maintenance target equipment 300 may be an altitude detection radar of an anti-aircraft guided weapon.

However, the type of the maintenance target equipment 300 described above is merely an exemplary description, and the maintenance target equipment 300 may be understood as a concept encompassing equipment utilized in various fields such as a general vehicle, an engine equipment, a transmission, and a generator. have.

According to one embodiment of the present disclosure, the component of the serviceable equipment 300 may include a circuit card.

The head mounted display apparatus 100 and the wearable apparatus 200 may be connected to each other through a network (not shown), and the network may mean a connection structure capable of exchanging information between nodes such as terminals and servers. Examples of such networks include 3rd Generation Partnership Project (3GPP) networks, Long Term Evolution (LTE) networks, 5G networks, World Interoperability for Microwave Access (WIMAX) networks, the Internet, and local area networks (LANs). , Wireless Local Area Network (WLAN), Wide Area Network (WAN), Personal Area Network (PAN), wifi network, Bluetooth (Bluetooth) network, satellite broadcasting network, analog broadcasting network, Digital Multimedia Broadcasting (DMB) network, etc. Included, but not limited to.

2 is a schematic diagram of a head mounted display device according to an exemplary embodiment of the present disclosure.

Referring to FIG. 2, the head mounted display apparatus 100 according to an exemplary embodiment of the present disclosure may include an image capturing unit 110, an object recognizing unit 120, an image correcting unit 130, and an output display unit 140. Can be.

The image capturing unit 110 may generate at least one area of the maintenance target device 300 to generate an input image.

According to an exemplary embodiment of the present application, the image capturing unit 110 may include a camera for generating an input image by capturing an image around the mechanic.

In addition, according to an exemplary embodiment of the present application, the image capturing unit 110 may generate the input image by a camera provided in the AR glass.

The object recognizing unit 120 obtains a spectral pattern associated with each of the plurality of parts based on the plurality of part-specific drawing data of the maintenance target device 300 and includes the maintenance included in the input image based on the spectral pattern. Identify parts of target equipment.

In addition, according to one embodiment of the present application, identifying the part of the maintenance target equipment, name of the part, assembly information including the part, location information occupied by the part in the assembly, within the assembly It can be understood to comprehensively grasp the functional information of the corresponding part.

For example, the object recognizer 120 may provide at least one of a result of identifying a part and durability information or wear information corresponding to the identified part.

In addition, according to an exemplary embodiment of the present disclosure, the head mounted display apparatus 100 may include a GPS or an orientation sensor, and the operator who wears the head mounted display apparatus 100 based on at least one of the GPS or orientation sensors. You can grasp eye contact information.

In addition, according to an exemplary embodiment of the present disclosure, the object recognition unit 120 may preferentially identify the parts included in the input image and viewed by the mechanic in association with the gaze information of the mechanic.

3 is a schematic diagram of an object recognition unit according to an exemplary embodiment of the present application.

Referring to FIG. 3, the object recognition unit 120 according to an exemplary embodiment of the present disclosure may include a point designation unit 121, a data logging unit 122, and a machine learning unit 123.

The point designation unit 121 receives the drawing data of the plurality of parts of the maintenance target equipment 300 including the plurality of parts, and designates a point specifying the shape of the part on the drawing data to perform modeling. Can be.

According to one embodiment of the present application, the point designation unit 121 is to receive the drawing data in association with software (3D S / W; for example, 3D Studio max, Maya, Rhino, etc.) relating to three-dimensional modeling Can be.

In addition, according to an embodiment of the present application, when the drawing data is the two-dimensional drawing data, the point designation unit 121 is a part associated with the two-dimensional drawing data from the two-dimensional drawing data using a three-dimensional modeling technique 3D drawing data can be generated.

In addition, according to an exemplary embodiment of the present application, the two-dimensional drawing data may include a plurality of image data photographed in various directions with respect to one component, and the point designation unit 121 analyzes the plurality of image data. The 3D drawing data may be generated.

Further, according to one embodiment of the present application, the 3D modeling technique may include a TL-embedding network technique, 3D Recurrent Reconstruction Neural Network (3D-R2N2) technique, MarrNet, etc., but is not limited thereto. Deep learning based three-dimensional modeling techniques can be applied.

In addition, according to the exemplary embodiment of the present application, the point designation unit 121 may designate a point for specifying the shape of the component using a Harris technique, a scale-invariant feature transform (SIFT) technique, or a SURF technique, but is not limited thereto. It is not.

In addition, according to one embodiment of the present application, the point specifying the shape of the part may be determined in association with the coordinate information or three-dimensional vector information on the virtual three-dimensional coordinate system.

The data logging unit 122 may derive the spectral information for each point by repeating the three-dimensional rotation on the result of the modeling.

According to the exemplary embodiment of the present application, the data logging unit 122 repeats random rotation on a three-dimensional coordinate system based on a modeling result of determining a point specifying a shape of a component derived by the point designation unit 121. Spectral information determined according to the change of the coordinate information of the point or the degree of change of the 3D vector information may be acquired by matching the degree of rotation (for example, the rotation direction and the angle) on the 3D coordinate system.

According to an embodiment of the present disclosure, the derivation of the spectral information by the data logging unit 122 repeats a process of deriving the correlation (function) between the degree of rotation on the 3D coordinate system and the spectral information based on the result of the modeling. For example, the relation may be described for all rotation directions to generate a plurality of log files including information on the relation.

The machine learning unit 123 may pattern the spectrum information by performing machine learning on the spectrum information.

In addition, according to one embodiment of the present application, the spectral pattern patterned by the spectral information, even if the operator looks at one part from various directions through machine learning based on the accumulation of the spectral information through repeated rotation, the spectral pattern Can be used to identify the part.

In addition, according to one embodiment of the present application, the machine learning unit 123 is a convolutional neural network (CNN), k-nearest neighbor algorithm (k-NN), random forest to pattern the spectral information Machine learning techniques such as Random Forest, adaptive boosting (AdaBoost), Linear Support Vector Machine (SVM), and Logistic Regression can be used. It is only an example, and should not be construed as limiting or limiting the application of other machine learning to the present idea.

The image corrector 130 may determine the resolution of the output image based on an AR glass standard, and determine the sharpness of the output image based on the illuminance of an external environment.

According to an exemplary embodiment of the present disclosure, since the operator's view of the external environment through the head mounted display apparatus 100 is often in a translucent form, the image corrector 130 may adjust the sharpness based on the illumination of the external environment. By making this decision, the operator can clearly identify the external environment.

The output display unit 140 may display an output image in which at least one of maintenance manual information of the corresponding component and the measurement result information overlap the input image, based on the identification result of the object recognition unit 120.

In addition, according to an embodiment of the present disclosure, the maintenance manual information may include at least one of repair accessory specification information, assembly procedure information, disassembly procedure information, and inspection reference information of the corresponding component.

In addition, according to one embodiment of the present application, the maintenance manual information, may include the maintenance know-how of the skilled person for the maintenance target equipment (300).

In addition, according to the exemplary embodiment of the present application, the output display unit 140 may amplify and output the output image on the display of the AR glass.

In addition, according to an exemplary embodiment of the present application, the output display unit 140 may implement and output the output image in the form of augmented reality, virtual reality, or mixed reality.

In addition, according to an exemplary embodiment of the present disclosure, the head mounted display apparatus 100 may include a GPS or an orientation sensor, and the operator who wears the head mounted display apparatus 100 based on at least one of the GPS or orientation sensors. You can grasp eye contact information.

In addition, according to an exemplary embodiment of the present disclosure, the output display unit 140 may include at least one of the maintenance manual information or the measurement result information on the input image with respect to the component currently viewed by the mechanic in connection with the eye gaze information of the mechanic. The overlapping output image may be displayed.

In addition, according to an exemplary embodiment of the present disclosure, the head mounted display apparatus 100 may select a part of which the operator wants to obtain at least one of the maintenance manual information and the measurement result information through a user input through a voice or gesture of a mechanic. It may include a separate user input receiving module (for example, a voice recognition module or a gesture recognition module).

In this case, according to the exemplary embodiment of the present application, the output display unit 140 may preferentially display at least one of maintenance manual information and measurement result information of the component selected by the operator based on the user input on the output image.

Figure 4 is a schematic diagram of a wearable device according to an embodiment of the present application.

Referring to FIG. 4, the wearable device 200 according to an exemplary embodiment of the present disclosure may include a measurement unit 210, a control unit 220, and a power supply unit 230.

The measurement unit 210 may be provided with an oscilloscope for measuring the electrical signal of the component to generate the measurement result information to generate the measurement result information.

According to an embodiment of the present application, the oscilloscope is in contact with a region of the part and performs an analog signal conversion or digital signal conversion on the probe unit for acquiring an electrical signal of the component and the electrical signal obtained by the pro unit as necessary. It may include a main body for generating a specific trigger signal.

According to an embodiment of the present disclosure, the probe unit may include a plurality of probes of a main probe and a sub probe, and the main probe and the sub probe may be used as necessary. It may be connected to the wearable device 200 by a separate wire or cable that can be expanded or contracted.

According to one embodiment of the present application, the tensile strength of the separate wire or cable may be preferably at least 500N.

In addition, according to one embodiment of the present application, the main probe and the sub probe may be implemented to be switched in accordance with a control signal of the control unit 220, whereby the maintenance of the electrical measurement of the component The main probe and the sub probe may be selected and utilized as necessary to facilitate the process.

In addition, according to an exemplary embodiment of the present disclosure, when the component is a circuit card, the measurement unit 210 measures an electrical signal of the circuit card to detect an abnormal waveform by comparing with a preset reference value, and the abnormal waveform is If detected, information indicating that the circuit card needs to be replaced may be included in the measurement result information.

In addition, according to one embodiment of the present application, the measurement unit 210 may be provided with a variety of sensors that can grasp the maintenance environment in the process of the maintenance or maintenance of the operator.

When the maintenance target equipment 300 corresponds to military equipment, the maintenance target equipment 300 may be operated without a defect even when a situation in which the maintenance target equipment 300 is to be operated such as a combat situation or an exhibition situation occurs suddenly. , Maintenance should be strictly compared to other equipment, maintenance target equipment (300) even in bad conditions such as external shock or vibration applied to the maintenance target equipment (300), rainy weather, night conditions, high temperature outside conditions The current state of a plurality of parts of the equipment under maintenance 300 (eg, in the case of a mating part, a tightly coupled or loose state, a threshold of external impact that the part can withstand, operating temperature range). Etc.), and it is necessary to know whether the parts that are electrically operated according to the external situation are operating normally.

According to an embodiment of the present disclosure, the measurement unit 210 may include a shock sensor, and may generate the measurement result information in association with external shock information of the maintenance environment identified through the shock sensor.

According to the exemplary embodiment of the present application, the measurement unit 210 may include a vibration sensor, and generate the measurement result information in connection with vibration information applied to the maintenance target device 300 identified through the vibration sensor. have.

According to one embodiment of the present application, the measurement unit 210 may include a temperature sensor, and may generate the measurement result information in connection with temperature information of the maintenance environment identified through the temperature sensor.

According to an embodiment of the present disclosure, the measurement unit 210 may include a humidity sensor, and may generate the measurement result information in connection with humidity information of the maintenance environment identified through the humidity sensor.

As described above, the measurement unit 210 is associated with various sensors to generate the measurement result information including the sensing signal of the sensor and the electrical signal of the component measured by the oscilloscope, the electrical signal measured by the oscilloscope The operator can see what conditions have been measured and can determine whether the part is operating properly under specific circumstances.

The controller 220 may include a communication module for transmitting the measurement result information to the head mounted display device and a GPU for controlling graphic processing of the output display unit.

According to an exemplary embodiment of the present disclosure, the controller 220 may include a controller associated with the AR glasses of the head mounted display apparatus 100.

According to an embodiment of the present disclosure, the control unit 220 is a single board computer (SBC) form in which one or more GPUs are mounted for smooth graphic processing (for example, an operation of displaying the output image) of the output display unit 140. It may be provided as.

According to an embodiment of the present disclosure, the controller 220 may be linked with an existing operating system such as Linux, Window, etc. in consideration of compatibility of the graphic library.

According to an embodiment of the present disclosure, the controller 220 may control the interface of the head mounted display apparatus 100 to perform channel switching as necessary.

According to the exemplary embodiment of the present application, the communication module refers to software (S / W) that operates to transmit the measurement result information of the measurement unit 210 to the head mounted display apparatus 100 so that a predetermined or less loss occurs. can do.

The power supply unit 230 may supply power to drive the measurement unit.

In addition, according to one embodiment of the present application, the power supply unit 230 may be to supply power for driving the measurement unit 210 and the control unit 220.

In addition, according to one embodiment of the present application, the power supply 230 may be provided with one or more lithium polymer battery.

In addition, according to one embodiment of the present application, the power supply 230 may be provided with a converter for providing a constant voltage, a separate charging circuit and the like.

As described above, the augmented reality-based maintenance system 10 using a wearable oscilloscope including a head-mounted display device 100 and a wearable device 200 that can be worn by a mechanic can measure electrical signals of equipment to be maintained. Maintenance manual information and measurement results of the electrical signal associated with the parts of the maintenance target equipment can be output based on the augmented reality to the device wearable by the operator.

Accordingly, even in the case of a non-skilled person in a narrow space as compared to the conventional maintenance system, the measurement, repair, maintenance work can be performed smoothly.

5 is an operation flowchart of an augmented reality-based maintenance method using a wearable oscilloscope according to an embodiment of the present application.

The augmented reality based maintenance method using the wearable oscilloscope illustrated in FIG. 5 may be performed by the augmented reality based maintenance system 10 using the wearable oscilloscope described above. Therefore, even if omitted below, the descriptions of the AR-based maintenance system 10 using the wearable oscilloscope may be equally applicable to the description of the AR-based maintenance method using the wearable oscilloscope.

Referring to FIG. 5, in operation S510, the head mounted display apparatus 100 may receive an input image of photographing at least one region of a maintenance target device including a plurality of components.

Next, in operation S520, the object recognition unit 120 may identify a part of the maintenance target equipment included in the input image based on the spectral pattern.

In addition, according to one embodiment of the present application, identifying the part of the maintenance target equipment, name of the part, assembly information including the part, location information occupied by the part in the assembly, within the assembly It can be understood to comprehensively grasp the functional information of the corresponding part.

Next, in operation S530, the measurement unit 210 may measure the electrical signal of the component through an oscilloscope to generate measurement result information.

In addition, according to an embodiment of the present application, the measuring unit 210 in step S530 is connected to a variety of sensors that can grasp the information of the maintenance environment, such as shock sensor, vibration sensor, temperature sensor, humidity sensor sensing signal of the various sensors Measurement result information can be generated, including.

Next, in operation S540, the output display unit 140 may display an output image in which at least one of the maintenance manual information of the identified part and the measurement result information overlap the input image.

In addition, according to one embodiment of the present application, the maintenance manual information may include at least one of repair accessory specification information, assembly procedure information, disassembly procedure information, inspection criteria information of the part.

Next, in operation S550, the image corrector 130 adjusts the resolution of the output image based on the standard of the head mounted display apparatus 100 (for example, the standard of AR glasses provided on the head mounted display apparatus). The brightness of the output image may be determined based on the illuminance of the external environment.

In the above description, steps S510 through S550 may be further divided into additional steps, or combined into fewer steps, according to embodiments herein. In addition, some steps may be omitted as necessary, and the order between the steps may be changed.

6 is a flowchart of a method of identifying a component of a serviceable device according to an embodiment of the present application.

The method of identifying the parts of the maintenance target equipment shown in FIG. 6 may be performed by the AR-based maintenance system 10 using the wearable oscilloscope described above. Therefore, even if omitted below, the descriptions of the AR-based maintenance system 10 using the wearable oscilloscope may be equally applicable to the description of the method of identifying the parts of the maintenance target equipment.

Referring to FIG. 6, in step S610, the point designation unit 121 may receive drawing data of the plurality of parts of the maintenance target device 300 including the plurality of parts.

According to an embodiment of the present application, in step S610, the point designation unit 121 is associated with the 2D drawing data from the 2D drawing data using a 3D modeling technique when the drawing data is 2D drawing data. Three-dimensional drawing data of parts can be generated.

Next, in step S620, the point designation unit 121 may perform modeling by designating a point specifying the shape of the part on the drawing data.

According to an embodiment of the present disclosure, in step S620, the point designation unit 121 may designate a point for specifying the shape of the part using a Harris technique, a scale-invariant feature transform (SIFT) technique, or a SURF technique.

Next, in step S630, the data logging unit 122 may derive the spectral information for each point by repeating the three-dimensional rotation on the result of the modeling.

According to one embodiment of the present application, the data logging unit 122 in step S630 the data logging unit 122 is a three-dimensional modeling result of the result of determining the point specifying the shape of the component derived by the point designation unit 121 By randomly repeating the rotation on the coordinate system, the spectrum information determined according to the degree of change of the coordinate information or the 3D vector information of the point can be obtained by matching the degree of rotation (for example, rotation direction and angle) on the 3D coordinate system. Can be.

Next, in operation S640, the machine learning unit 123 may perform machine learning on the spectrum information to pattern the spectrum information to obtain a spectrum pattern.

According to one embodiment of the present application, in step S640 the machine learning unit 123, Convolutional Neural Networks (CNN), k- nearest neighbor algorithm (k-NN), in order to pattern the spectral information, Machine learning techniques such as Random Forest, adaptive boosting (AdaBoost), Linear Support Vector Machine (SVM), and Logistic Regression can be utilized.

Next, in operation S650, the object recognition unit 120 may identify a part of the maintenance target equipment included in the input image based on the spectral pattern.

In the above description, steps S610 to S650 may be further divided into additional steps, or combined into fewer steps, according to embodiments herein. In addition, some steps may be omitted as necessary, and the order between the steps may be changed.

7 is a flowchart illustrating a method of generating measurement result information by measuring an electrical signal of a component of a maintenance target device according to an exemplary embodiment of the present application.

The method of generating the measurement result information by measuring the electrical signal of the component of the maintenance target equipment shown in FIG. 7 may be performed by the AR-based maintenance system 10 using the wearable oscilloscope described above. Therefore, even if omitted below, the description of the augmented reality-based maintenance system 10 using the wearable oscilloscope is the same as the description of the method for generating the measurement result information by measuring the electrical signal of the parts of the equipment to be repaired. Can be applied.

Referring to FIG. 7, in operation S710, when the component is a circuit card, the measurement unit 210 may measure an electrical signal of the circuit card and detect an abnormal waveform by comparing with a preset reference value.

Next, in operation S720, when the abnormal waveform is detected, the measurement unit 210 may include information indicating that the circuit card needs to be replaced in the measurement result information.

In the above description, steps S710 to S720 may be further divided into additional steps, or combined into fewer steps, according to embodiments herein. In addition, some steps may be omitted as necessary, and the order between the steps may be changed.

Augmented reality-based maintenance method using a wearable oscilloscope according to an embodiment of the present application may be implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the augmented reality-based maintenance method using the wearable oscilloscope described above may be implemented in the form of a computer program or an application executed by a computer stored in a recording medium.

The above description of the present application is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

10: Augmented Reality Based Maintenance System Using Wearable Oscilloscope
100: head mounted display device
110: video recording unit
120: object recognition unit
121: point designator
122: data logging section
123: machine learning unit
130: image correction unit
140: output display unit
200: wearable device
210: measurement unit
220: control unit
230: power supply

Claims (15)

In augmented reality based maintenance system using a wearable oscilloscope,
A head mounted display device for identifying equipment to be repaired including a plurality of parts and providing maintenance measures information to a technician based on a state of the equipment to be repaired; And
Including a wearable device for measuring the state of the maintenance target equipment to generate the measurement result information,
The head mounted display device,
An image photographing unit photographing at least one region of the maintenance target equipment to generate an input image;
Object recognition for acquiring a spectral pattern associated with each of the plurality of parts based on the plurality of part-specific drawing data of the maintenance target equipment and identifying a part of the maintenance target equipment included in the input image based on the spectral pattern part;
An output display unit configured to display an output image in which at least one of maintenance manual information of the corresponding component and the measurement result information overlap the input image based on the identification result; And
A user input receiving module for receiving a user input by voice or gesture so that a mechanic can select a part to obtain the maintenance manual information;
Including,
The output display unit,
When the user input is received, the maintenance manual information of the parts selected by the operator based on the user input is preferentially displayed on the output image,
Characterized in that the sensing information for the corresponding component is included in the output image and outputted;
The wearable device,
A measurement unit having an oscilloscope for measuring the electrical signal of the part and generating the measurement result information;
A control unit including a communication module for transmitting the measurement result information to the head mounted display device and a GPU for controlling graphic processing of the output display unit; And
It includes a power supply for supplying power for driving the measurement unit,
The measurement unit,
At least one of a shock sensor, a vibration sensor, a temperature sensor, and a humidity sensor,
The measurement result information,
And the sensing information associated with the selected part based on the user input, wherein the sensing information is applied to external impact information of the maintenance environment identified through the shock sensor and the maintenance target device identified through the vibration sensor. Generated to include at least one of vibration information, temperature information of the maintenance environment identified through the temperature sensor, and humidity information of the maintenance environment identified through the humidity sensor,
The oscilloscope,
A probe portion in contact with a portion of the component and obtaining an electrical signal of the component,
The probe unit includes a main probe and a sub probe connected to the wearable device, respectively, by a wire or a cable, wherein the main probe and the sub probe are implemented to be switched according to a control signal of the controller. Infrastructure maintenance system. The method of claim 1,
The object recognition unit,
A point designation unit configured to receive drawing data of the plurality of parts of the maintenance target equipment including the plurality of parts, designate a point specifying a shape of the part on the drawing data, and perform modeling;
A data logging unit for deriving spectral information for each point by repeating a three-dimensional rotation on a modeling result; And
A machine learning unit which performs machine learning on the spectrum information to pattern the spectrum information;
To include, augmented reality-based maintenance system. The method of claim 1,
The head mounted display device includes an AR glass,
The image photographing unit generates the input image by a camera provided in the AR glass,
The output display unit is to augment the output image on the display of the AR glass, augmented reality-based maintenance system. The method of claim 3,
The head mounted display device,
And an image corrector configured to determine the resolution of the output image based on the standard of the AR glasses and determine the sharpness of the output image based on illuminance of an external environment. The method of claim 1,
The component comprises a circuit card,
The measurement unit,
Measuring the electrical signal of the circuit card to detect the abnormal waveform through comparison with a predetermined reference value, and if the abnormal waveform is detected, the measurement result information that includes information that the circuit card needs to be replaced ,
Augmented reality based maintenance system. The method of claim 1,
The head mounted display device is formed in the form of a helmet or band that can be worn on the head of the mechanic,
The wearable device is formed in the form of a vest or jacket wearable on the body of the mechanic, augmented reality-based maintenance system. The method of claim 1,
The maintenance target equipment is an altitude detection radar of anti-aircraft guided weapons,
Augmented reality based maintenance system. The method of claim 1,
The maintenance manual information mentioned above,
Characterized in that at least one of the repair accessory specification information, assembly procedure information, disassembly procedure information, inspection criteria information of the corresponding parts,
Augmented reality based maintenance system. delete delete delete delete delete delete delete

Images