KR20210113810A - Augmented reality based maintenance manual providing method and device for construction machinery - Google Patents

Abstract

The present invention relates to a method for providing a construction machinery maintenance manual and provides an augmented reality content selection menu for the provision of a construction machinery maintenance manual. A user's content selection signal is received. A virtual 3D model corresponding to the construction machinery is augmented on a display unit in response to the content selection signal. Augmented reality maintenance manual information is displayed on a corresponding position of the 3D model according to the content selection signal.

Description

A method and apparatus for providing maintenance manuals for construction machinery using augmented reality

The present invention relates to a method and apparatus for providing a maintenance manual for a construction machine. More specifically, it relates to a method and apparatus for providing a maintenance manual of a construction machine using augmented reality.

There are many difficulties and inconveniences in checking various information at the site during performance testing and maintenance of construction equipment. In addition, since it is difficult for non-experts to directly perform, so many measures are taken through an on-site dispatch of experts, it takes a lot of time and money, and there is a problem in that the non-operation time of the equipment increases and the loss of the customer also increases.

One object of the present invention is to provide a method for providing information on the performance test and maintenance of a construction machine using augmented reality.

Another object of the present invention is to provide a portable device for performing the above-described method.

In the method of providing a maintenance manual for a construction machine according to exemplary embodiments for achieving the object of the present invention, an augmented reality content selection menu for providing a maintenance manual for a construction machine is provided. Receive a user's content selection signal. A virtual 3D model corresponding to the construction machine is augmented on the display unit in response to the content selection signal. Augmented reality maintenance manual information is displayed at the corresponding position of the 3D model according to the content selection signal.

In example embodiments, receiving the user's content selection signal may include receiving a performance test guide request signal or a maintenance guide request signal, and receiving a detailed item request signal.

In example embodiments, augmenting the 3D model on the display unit displays an image collected through a camera unit as an image on the display unit, provides the 3D model on the image, and is selected by a user placing the 3D model in a location.

In example embodiments, providing the 3D model may include providing the 3D model on a ground close to the user on the display unit.

In example embodiments, displaying the augmented reality guide information at the corresponding position of the 3D model displays detailed parts of the 3D model to be diagnosed according to the content selection signal, and the details of the 3D model may include displaying maintenance manual information associated with the part.

In example embodiments, displaying the detailed part of the 3D model includes moving to an internal location of the 3D model in response to a user input signal to provide the shape of the detailed part at the internal location can do.

In example embodiments, displaying the detailed part of the 3D model provides that the external shape of the 3D model has a first transparency, and the detailed part has a second transparency that is lower than the first transparency This may include providing

In example embodiments, the augmented reality guide information may include manual information for performing a performance test or maintenance of the construction machine.

A portable device according to exemplary embodiments for achieving another object of the present invention includes a camera unit for collecting images, a display unit for displaying images collected through the camera unit, and augmentation for providing maintenance manuals for construction machines A control unit for augmenting the virtual 3D model corresponding to the construction machine on the display unit in response to the real content selection signal and displaying the augmented reality maintenance manual information at the corresponding position of the 3D model according to the augmented reality content selection signal include

In example embodiments, the controller displays the image collected through the camera unit as an image on the display unit, provides the 3D model on the image, and arranges the 3D model at a location selected by the user. can

In example embodiments, the control unit may provide the 3D model on the ground close to the user on the display unit.

In example embodiments, the controller may display a detailed part of the 3D model to be diagnosed according to the content selection signal, and display diagnostic guide information related to the detailed part of the 3D model.

In example embodiments, the controller may move to an internal position of the 3D model in response to a user input signal to provide the shape of the detailed part at the internal position.

In example embodiments, the control unit may provide the external shape of the 3D model to have a first transparency, and provide the detailed part to have a second transparency lower than the first transparency.

In example embodiments, the augmented reality maintenance manual information may include maintenance manual information for performing a performance test or maintenance of the construction machine.

According to exemplary embodiments, information on detailed performance items and methods for performance testing and maintenance can be provided easily and quickly and intuitively by utilizing augmented reality technology. Therefore, it is possible to improve the operator's diagnostic capability and self-maintenance capability, and to increase the operating time of equipment and reduce maintenance costs through rapid maintenance.

However, the effects of the present invention are not limited to the above-mentioned effects, and may be variously expanded without departing from the spirit and scope of the present invention.

1 is a block diagram illustrating a portable device according to example embodiments.
2 is a flowchart illustrating a method of providing a maintenance manual for a construction machine using augmented reality according to exemplary embodiments.
3 is a flowchart illustrating an augmented reality content providing step (or augmented reality content providing step for a maintenance guide) for a performance test guide in the maintenance manual providing method of FIG. 2 .
FIG. 4 is a view showing a start screen of an application for performing the method of providing a maintenance manual of a construction machine of FIG. 2 .
5 to 11 are diagrams for explaining an example of a step of providing augmented reality content for the performance test guide of FIG. 3 .
12 to 18 are diagrams for explaining an example of the step of providing augmented reality content for the maintenance guide of FIG. 3 .

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In each drawing of the present invention, the dimensions of the structures are enlarged than the actual size for clarity of the present invention.

In the present invention, terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

The terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

With respect to the embodiments of the present invention disclosed in the text, specific structural or functional descriptions are only exemplified for the purpose of describing the embodiments of the present invention, and the embodiments of the present invention may be embodied in various forms. It should not be construed as being limited to the embodiments described in

That is, since the present invention can have various changes and can have various forms, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

1 is a block diagram illustrating a portable device according to example embodiments.

Referring to FIG. 1 , the portable device 100 includes a communication unit 110 , an input unit 120 , a camera unit 130 , a sensor unit 140 , a control unit 150 , a storage unit 160 , and a display unit 170 . may include

In example embodiments, the portable device 100 may be a terminal for providing augmented reality content. The portable apparatus 100 may include a wearable device such as a smart phone, a tablet, a glass-type terminal, and a watch-type terminal used by a user. Here, the user may mean a person who inputs a signal to the portable device and uses an augmented reality application program.

The communication unit 110 may perform data communication with the outside. The communication unit 110 may include a transmitter for up-converting and amplifying a frequency of a transmitted signal, and a receiver for low-noise amplifying a received signal and down-converting a frequency of the received signal.

The communication unit 110 may communicate with a server that provides 3D model information as a 3D computer graphic image of a construction machine to be diagnosed and augmented reality maintenance manual information for a diagnosis guide of the construction machine. The communication unit 110 may transmit a 3D model information request, an augmented reality maintenance manual information request, and the like to the server. In addition, the communication unit 110 may receive the 3D model information, the augmented reality maintenance manual information, and the like from the server.

The input unit 120 may detect a user's input and transmit an input signal corresponding to the user's input to the control unit 130 . The input unit 120 may include a button, a jog dial, a wheel, and the like, and may detect a user's human body or a pen input using a touch sensor, an electromagnetic sensor, a proximity sensor, an IR sensor, or the like.

The input unit 120 may be configured as an input pad. The input unit 120 may be configured to mount various sensors on an input pad. The input unit 120 may be configured as an input pad in which a touch sensor is attached to the input pad in the form of a film or combined in the form of a panel. The input unit 120 may be configured as an EMR (Electro Magnetic Resonance) or EMI (Eletro Magnetic Interference) type input pad using an electromagnetic sensor. The input unit 120 may include one or more input pads forming a mutually layered structure in order to sense an input using a plurality of sensors.

The input unit 120 forms a layer structure with the display unit 170 to operate as an input screen. For example, the input unit 120 may include an input pad having a touch sensor and may include a touch screen panel (TSP) coupled to the display unit 170 .

For example, the input unit 120 may detect a request for arrangement and movement of the 3D model, a request for display and selection of augmented reality content, and the like.

The camera unit 130 may collect an image including at least one subject, convert it into a data signal, and transmit it to the controller 150 . The controller 150 may control the display unit 170 to recognize a specific subject from the image or display the collected image as a preview screen based on the data signal received from the camera unit 130 . Also, when a request to perform a photographing is detected, the controller 150 may generate an image from the image collected through the camera unit 130 at the point in time when the request to perform a photographing is detected. The controller 150 may temporarily or permanently store the generated image in the storage 160 .

The sensor unit 140 may include a plurality of sensors for detecting the current state of the portable device 100 . The sensor unit 140 may generate a detection signal corresponding to the sensed state and transmit it to the control unit 150 .

For example, the sensor unit 140 may include a GPS sensor 142 , a gravity sensor 144 , and a gyro sensor 146 to collect geographic/location information of the portable device 100 .

The GPS sensor 142 generates current location information of the portable device 100 based on the GPS signal received from the GPS satellite through the communication unit 110 . The GPS sensor 142 may determine the latitude, longitude, and altitude of the portable device 100 by using a GPS signal received from a plurality of GPS satellites, for example, using triangulation.

The gravity sensor 144 detects in which direction gravity acts. The gravity sensor 144 detects a gradient of the electronic device 100 by detecting gravity acting on the electronic device 100 , that is, a direction of acceleration, or detects geomagnetism around the device 100 , and the electronic device 100 . direction can be determined. Since the gravity sensor 144 determines the orientation using geomagnetism, it may be configured as a geomagnetic sensor or may operate similarly to a geomagnetic sensor.

The gyro sensor 146 is a sensor that measures the rotation angle of the portable device 100 and may detect the rotation angle with respect to the 3D reference axis. For example, the gyro sensor 146 detects an angle rotated with respect to the x, y, and z axes with the center of the portable device 100 as the origin, ie, pitch, roll, and yaw. can do.

The control unit 150 may control the communication unit 110 to transmit information obtained through the sensor unit 140 to the server, and the server may provide augmented reality maintenance manual information based on the information. In addition, the controller 150 controls the display unit 170 to move the 3D model on the display unit 170 according to the motion detection information according to the movement of the portable device 100 acquired through the sensor unit 140 , It is possible to allow the user to view various positions inside and outside the 3D model.

The controller 150 may control each component for the overall operation of the portable device 100 . For example, the controller 150 may control each component of the portable device 100 to provide augmented reality content for a diagnosis guide.

For example, the controller 150 may control the display unit 170 to display the augmented reality content in response to a display request for the augmented reality content.

A more specific operation of the controller 150 will be described in detail below with reference to the drawings.

The storage unit 160 may store programs or commands for the portable device 100 . The controller 150 may execute programs or commands stored in the storage 160 .

The storage unit 160 includes a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, It may include at least one type of storage medium among a magnetic disk and an optical disk.

For example, the storage unit 160 may store at least one augmented reality content.

The display unit 170 displays (outputs) information processed by the portable device 100 . For example, the display unit 170 may display information corresponding to a currently running application, program, or service together with a user interface (UI).

The display unit 170 is a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display (flexible display) ), and at least one of a three-dimensional display (3D display).

The display unit 170 may operate as a touch screen by forming a layer structure with a touch sensor and/or an electromagnetic sensor constituting the input unit 120 . In this case, the display unit 170 operating as a touch screen may perform a function of an input device.

The display unit 170 may display an augmented reality content list or augmented reality content under the control of the controller 150 .

Since the components shown in FIG. 1 are not essential, the portable device 100 having more or fewer components may be implemented.

Hereinafter, a method of providing a maintenance manual for a construction machine using augmented reality using the portable device of FIG. 1 will be described.

2 is a flowchart illustrating a method of providing a maintenance manual for a construction machine using augmented reality according to exemplary embodiments. 3 is a flowchart illustrating an augmented reality content providing step (or augmented reality content providing step for a maintenance guide) for a performance test guide among the maintenance manual providing method of FIG. 2 . FIG. 4 is a view showing a start screen of an application for performing the method of providing a maintenance manual of a construction machine of FIG. 2 .

2 and 3 , when the augmented reality application is executed, an augmented reality content selection menu for a diagnostic guide of a construction machine may be provided (S200), and a user's content selection signal may be received (S210).

As shown in FIG. 4 , in exemplary embodiments, when the augmented reality application is started, the controller 150 may provide a selection menu on the display unit 170 of the portable device 100 . . A selection button for selecting a performance test guide or a maintenance guide of a construction machine may be provided in the selection menu.

Next, the controller 150 may determine whether the augmented reality content request is detected. The control unit 150 may receive a performance test guide request signal or a maintenance guide request signal input through the input unit 120 .

Subsequently, it is possible to provide an augmented reality overlay image including augmented reality content according to the performance test guide request signal or the maintenance guide request signal (S220, S230).

First, the control unit 150 provides a detailed item check list on the display unit 170 of the portable device 100 (S222 or S232), receives a user's selection signal of the detailed item (S224 or S232), and selects the Augmenting the virtual 3D model corresponding to the construction machine on the display unit 170 in response to the signal (S226 or S236), and displaying the augmented reality maintenance manual information at the corresponding position of the 3D model according to the selection signal There is (S228 or S238).

The augmented reality overlay image may include three virtual models augmented on the display unit 170 and an image of maintenance manual information superimposed on the 3D model. The image of the maintenance manual information may be displayed at a corresponding position of the 3D model by a template. The template may include information on a location to associate a context representing the content of the information to the 3D model.

In exemplary embodiments, after providing the augmented reality content information for the performance test guide, when the user requests the maintenance guide, the control unit 150 may provide augmented reality content information for the maintenance guide have.

Hereinafter, an example of the step of providing augmented reality content for the performance test guide will be described.

5 to 11 are diagrams for explaining an example of an augmented reality content providing step for the performance test guide of FIG. 3 .

As shown in FIG. 5 , when the performance test guide is selected, a performance test checklist may be provided. For example, in the performance test check list, a selection button for selecting the hydraulic cylinder cycle time, turning function deviation check, track rotation speed, track deviation check, turning speed, traveling speed, or cylinder settlement measurement items may be provided.

When the hydraulic cylinder cycle time item is selected, a virtual 3D model corresponding to the construction machine may be augmented on the display unit 170 .

First, as shown in FIGS. 6 and 7 , the control unit 150 displays the image collected through the camera unit 130 as an image on the display unit 170 , and the construction machine model (DX380LC-) on the image. A 3D model M corresponding to 5) may be provided.

When the camera unit 130 moves to photograph the ground close to the user, the 3D model M is provided on the target C displayed on the display unit 170, and when the user taps the screen, it is arranged at the displayed position, and the The 3D model M may be displayed based on the x, y, and z coordinates stored on the screen.

The 3D model M of FIG. 7 may be moved by a user's input signal. For example, the 3D model M may be zoomed in or out by making the distance between two fingers touched on the screen closer or farther away. By sliding two fingers, the 3D model M may be rotated, and by sliding one finger in one direction, the 3D model M may be moved in one direction. Also, by moving the portable device 100 (according to motion detection information according to the motion of the portable device 100 ), the 3D model M may be displayed in a different direction.

In example embodiments, in response to the user's input signal, the 3D model M may move to an internal position to provide the shape of the detailed part at the internal position.

As shown in FIGS. 8 and 9 , a virtual image of the inside of the cabin may be provided by inputting a move inside selection button (Go Inside) provided on the screen or by a user input signal.

As shown in FIG. 10 , when the user selects the boom lowering speed of the boom cylinder, an animation of the boom cylinder descending may be produced, and the performance standard time may be displayed.

Accordingly, the user may be provided with guide information for performance testing of detailed parts such as the boom cylinder. The user can be provided with information on the performance test method for the arm cylinder and bucket cylinder. In addition, information on a joystick operation method for the performance test, test conditions (power mode, engine control dial, work mode, auto idle, etc.) may be provided.

Accordingly, the user experiences the content information for the performance test of the detailed parts of the construction machine in the form of a step-by-step scenario, thereby providing an environment in which non-specialists can perform the performance test through augmented reality.

11 , when the selected detailed item is finished, a performance test checklist is provided, and the user can proceed to the next item. In addition, a selection button for selecting the maintenance guide is provided, and the user can proceed directly to the step of providing augmented reality content for the maintenance guide.

Hereinafter, an example of the step of providing augmented reality content for the maintenance guide will be described.

12 to 18 are diagrams for explaining an example of an augmented reality content providing step for the maintenance guide of FIG. 3 .

In exemplary embodiments, when the maintenance guide is selected, a detailed maintenance checklist may be provided. As a result of the performance test using the performance test guide, when an issue related to the arm crowd occurs, a detailed maintenance item related to the arm crowd may be selected.

12 , when a maintenance item related to the arm crowd is selected from among the detailed items of the maintenance guide, a detailed test check list may be provided. For example, the detailed test check list includes a selection button for selecting the items to check the arm crowd pilot pressure and pressure sensor, check the arm spool pilot line orifice, check the arm holding valve, check the ORV (Overload Relief Valve), or check the arm regeneration valve. This can be provided.

13 and 14, when the arm crowd pilot pressure and pressure sensor check item is selected, a virtual 3D model (M) corresponding to the construction machine model (DX380LC-5) is displayed on the display unit 170 can be augmented. In this case, detailed parts corresponding to the arm crowd pilot pressure and pressure sensor may be displayed together. For example, the controller 150 may display the external shape of the 3D model M in a perspective view having a first transparency, and display the detailed parts to have a second transparency lower than the first transparency.

As shown in FIG. 15 , the 3D model M may be enlarged by the user's input signal to provide an image of the internal detailed parts. P1 and P2 of FIG. 15 may represent arm crowd pilot pressures during arm crowd operation, and S may represent a pressure sensor.

As shown in FIG. 16 , when the user selects to view the circuit diagram, a circuit diagram showing the arm crowd pilot pressures and the pressure sensor may be provided during the arm crowd operation.

Although not shown in the drawing, when the user selects to view the performance standard value, a performance standard value graph or numerical value may be provided.

When the Next button is pressed on the screen of FIG. 14 , a maintenance guide phrase is provided at the bottom of the screen of FIG. 17 , and when the Next button is pressed on the screen of FIG. 17 , a maintenance guide phrase can be provided at the bottom of the screen of FIG. 18 .

Therefore, the user can check the position of inspection and maintenance of the arm circuit by checking the hydraulic flow of the high pressure line, pilot and return line connected to the actuator from the hydraulic pump through the main control valve in relation to the arm crowd issue through the augmented 3D model. easily recognizable.

In addition, the user can check whether there is an abnormality simply by comparing it to the DMS monitor derived using the standard graph for the arm crowd performance, check the pilot pressure and check the position of the arm crowd pilot sensor, replace the connector and check the arm crowd pilot sensor Guidance on replacement can be provided.

If the same abnormal symptom occurs even after the measures according to the maintenance guide of the cancer crowd item, you can request A/S.

As described above, information on detailed performance items and methods for performance testing and maintenance can be provided easily and quickly and intuitively by utilizing augmented reality technology. Therefore, it is possible to improve the operator's diagnostic capability and self-maintenance capability, and to increase the operating time of equipment and reduce maintenance costs through rapid maintenance.

Although the above has been described with reference to the embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below. You will understand that you can.

100: mobile device 110: communication unit
120: input unit 130: camera unit
140: sensor unit 150: control unit
160: storage unit 170: display unit

Claims (15)

providing an augmented reality content selection menu for providing maintenance manuals for construction machinery;
receive a content selection signal of the user;
augmenting a virtual 3D model corresponding to the construction machine on a display unit in response to the content selection signal; and
A method of providing maintenance manuals for construction machines, comprising displaying augmented reality maintenance manual information at the corresponding position of the 3D model according to the content selection signal. The method of claim 1, wherein receiving the user's content selection signal comprises:
receive a performance test guide request signal or a maintenance guide request signal; and
A method of providing a maintenance manual for a construction machine, comprising receiving a detailed item request signal. The method of claim 1 , wherein augmenting the 3D model on the display unit comprises:
displaying the image collected through the camera unit as an image on the display unit;
providing the 3D model on the image; and
A method of providing a maintenance manual for a construction machine, comprising arranging the 3D model at a location selected by a user. The method of claim 3 , wherein the providing of the 3D model comprises providing the 3D model on the ground close to the user on the display unit. According to claim 1, Displaying the augmented reality maintenance manual information in the corresponding position of the 3D model
displaying detailed parts of the 3D model to be diagnosed according to the content selection signal; and
A method of providing a maintenance manual for a construction machine, comprising displaying maintenance manual information related to the detailed parts of the 3D model. The method of claim 5 , wherein displaying the detailed parts of the 3D model comprises:
A method of providing a maintenance manual for a construction machine, comprising moving to an internal position of the 3D model in response to a user's input signal and providing the shape of the detailed part at the internal position. The method of claim 5 , wherein displaying the detailed parts of the 3D model comprises:
providing that the external shape of the 3D model has a first transparency; and
The method of providing a maintenance manual for construction machinery comprising providing the detailed parts to have a second transparency lower than the first transparency. The method of claim 1 , wherein the augmented reality maintenance manual information includes guide information for performing a performance test or maintenance of the construction machine. a camera unit for collecting images;
a display unit for displaying images collected through the camera unit; and
In response to an augmented reality content selection signal for providing a maintenance manual of a construction machine, a virtual 3D model corresponding to the construction machine is augmented on the display unit, and augmented at a corresponding position of the 3D model according to the augmented reality content selection signal A portable device including a control unit for displaying information about the actual maintenance manual. The portable device of claim 9 , wherein the controller displays the image collected through the camera unit as an image on the display unit, provides the 3D model on the image, and arranges the 3D model at a location selected by a user. . The portable device of claim 10 , wherein the control unit provides the 3D model on a ground close to the user on the display unit. The portable device of claim 9 , wherein the controller displays detailed parts of the 3D model to be diagnosed according to the content selection signal, and displays maintenance manual information related to the detailed parts of the 3D model. The portable device of claim 12 , wherein the controller moves to an internal position of the 3D model in response to a user input signal to provide the shape of the detailed part at the internal position. The portable device of claim 12 , wherein the control unit provides an external shape of the 3D model to have a first transparency, and provides the detailed part to have a second transparency lower than the first transparency. The portable device of claim 9, wherein the augmented reality maintenance manual information includes guide information for performing a performance test or maintenance of the construction machine.

Images