KR102174035B1 - Object inspection method using an augmented-reality - Google Patents

Abstract

The present invention includes a first step of extracting feature point data of an object; A second step of driving an augmented reality app after the first step; A third step of tracking the object after the second step; A fourth step of setting an assembly position of an object part after tracking the object after the third step; A fifth step of detecting whether a component is assembled after the fourth step; A sixth step of determining whether a component assembly error has occurred after the fifth step; A seventh step of projecting the augmented reality content onto the object image of the terminal after the sixth step; An object inspection method using augmented reality including an eighth step of notifying an operator when an assembly mistake is confirmed after the seventh step may be provided.

Description

Object inspection method using an augmented-reality}

The present invention relates to an object inspection method using augmented reality.

Since minor mistakes in various work sites can be directly linked to defects in products, various methods are used to help workers work and prevent mistakes.

In the case of assembly of parts, a pickup (pick-up) optical sensor or a vision sensor is generally used.In the former case, a sensor must be installed by wire for each component box, and in the latter case, the equipment is expensive and It is not easy to detect assembly.

Devices that track the position of an operator's hand using an ultrasonic sensor and support assembling parts at a predetermined position are also being announced, but the device is relatively complex and its performance is not sufficiently verified.

The present invention provides a method for inspecting an object using augmented reality to assist in the assembly of parts of an object and to detect an assembly error in real time.

Extracting feature point data of the object;

Driving an augmented reality app;

Tracking the object;

Setting a component assembly position after tracking the object;

Detecting whether a component is assembled;

Projecting augmented reality content onto the object image;

Notifying the operator when an assembly mistake is confirmed;

An object inspection method using an augmented reality including a may be provided.

The present invention applies an augmented reality (AR) technique that has been mainly used in the fields of navigation and education to the industrial field. By projecting virtual contents for work support on the actual work site, it is possible to realize ubiquitous work support.

Since an application using augmented reality is provided, it is possible to prevent mistakes that occur, for example, when assembling vehicle parts.

According to the present invention, it is possible to provide a virtual reality-based mobile Android app to assist component assembly work and detect assembly mistakes in real time.

The present invention can detect in real time a component assembly error and notify an operator when assembling a component of an object through augmented reality.

Since it is possible to check component assembly mistakes in real time, it is possible to significantly reduce the defect rate of the product.

1 is a flowchart showing an object inspection method using augmented reality according to the present invention.
2 is a view showing the shape of the object and the assembly position of the parts.
3 is an explanatory diagram showing a process of extracting a part assembly position (POIt) of an object.
4 is a diagram showing augmented reality content for supporting assembly of parts of an object.
5 is a diagram showing a red mark on an object.
6 to 9 are views showing an image confirming whether an assembly error occurs when assembling an object part.

Hereinafter, specific details for carrying out the present invention will be described in detail based on the accompanying exemplary drawings.

The present invention is a method for inspecting an object using augmented reality to assist in the assembly work of parts of the object and to detect assembly errors in real time.

1 is a flowchart showing an object inspection method using augmented reality according to the present invention.

Referring to Figure 1, the inspection method of the present invention is a pre-processing step of extracting feature point data of an object, a first step (S1), a second step of driving an augmented reality app (S2), a third step of tracking the object ( S3), the fourth step of setting the assembly position of the target part after tracking the target (S4), the fifth step of detecting whether or not the parts are assembled (S5), the sixth step of determining whether or not the parts are assembled (S6), the object of the terminal It may include a seventh step (S7) of projecting the augmented reality content onto the image, and an eighth step (S8) of notifying the operator by voice when an assembly mistake is confirmed.

In the process of moving from the third step (S3) of tracking the object to the fourth step (S4) of setting the parts assembly position, if the augmented reality app has not determined the part assembly position setting, the fourth step (S4) You can move to and set a new location for assembling the parts of the object.

In the process of moving from the third step (S3) of tracking the object to the fourth step (S4) of setting the part assembly position, if the augmented reality app has already confirmed the part assembly position setting, the part assembly position is newly set. The fourth step may be omitted and proceed to a fifth step (S5) of detecting whether or not the parts are assembled.

As a result of the determination in the fifth step (S5) of detecting whether the parts are assembled, the augmented reality content may be projected onto the object image of the terminal in the seventh step (S7).

If the assembly of the parts is wrong, the eighth step (S8) of notifying the operator by voice may be performed.

It may include a ninth step (S9) of performing the third step (S3) by returning to the second step (S2) after the eighth step (S8).

An embodiment of the present invention will be described with reference to FIGS. 2 to 3.

As an embodiment of the present invention, an example applied to assembly of parts of the vehicle valve body 10 as an object will be described as follows.

In the first step (S1), a 2D valve body image (target image) is generated by taking a picture of the commercial vehicle valve body 10 that is a component assembly target with a camera of a terminal, for example, a smartphone, and tracking the target image in real time. A data set of feature points to be used can be extracted.

The feature point extraction process performed in the first step S1 may be performed as a pre-processing process. That is, it can be performed only once offline (off-line) before running the app.

After the augmented reality app is executed in the second step (S2), it is possible to set a component assembly position for each object repeatedly input one by one, and determine a component assembly error for the object.

After installing the camera of the smartphone at the position where the target valve body 10 is visible, the augmented reality app is driven, and based on the feature point data set obtained in the first step of the preprocessing process, the pose of the target valve body can be found. I can. Since several objects may be sequentially input, the object tracking step of the third step (S3) of finding a pose for each newly introduced object may be repeatedly performed. The object tracking step may be continuously repeated for each object while the augmented reality app is running.

In the third step, if the pose of the object, which is information including the posture or position of the object relative to the camera, is known, it is necessary to know the assembly position of the part within the object. The parts assembly location we need to know is shown in FIG. 2. It is necessary to know the target coordinates and touch coordinates of each part assembly location. It is necessary to know the target seat POIt1 of the first part assembly position of the first part part1, and also the touch coordinate POIc of the first part assembly position.

On the other hand, it is necessary to project augmented reality content, which is information including the assembly position of parts, on the image of the valve body, which is an object photographed by the smartphone. In order to project augmented reality content, it is necessary to know the assembly position of the parts in advance. Since the position data of each part is not given in the image of the valve body as the object, for example, it is necessary to determine the assembly positions of five parts. This is the fourth step.

3 illustrates the fourth step. A description will be made showing the extraction process of the assembly position of the part of the valve body as an object. Referring to FIG. 3, when the app is executed and the target valve body is successfully tracked, a target pose matrix may be generated. After obtaining a model-view matrix expressed based on the camera coordinate system from the target pose matrix, the model-view matrix can be stored together with a perspective projection matrix.

The perspective-projection matrix can be viewed as expressing the camera's installation function with respect to the ground at which position or angle the camera is installed with respect to the ground.

The model-view matrix can be viewed as expressing the loading function of the object relative to the ground, at which position on the ground and at what angle, when the camera is fixed at a specific angle on the ground.

After the object tracking is complete, if the user finds and touches the assembly position of the part from the object image displayed on the smartphone screen, the touch coordinates of each part assembly position can be generated, and the app extracts the touch coordinates (POIc) on the screen. Thereafter, inverse matrices of the previously stored model-view matrix and perspective-projection matrix may be obtained, and target coordinates (POIs) corresponding to touch coordinates may be calculated using the inverse matrices. The target coordinates may be coordinates of each component assembly position in the object coordinate system based on the object.

In a first step, which is a pre-processing step, a feature point of a valve body as an object is extracted, and the feature point may be, for example, an outline of the valve body or a boundary line between the ground and the valve die. In addition, the feature point may be a hole in the valve body or an edge of a protruding portion. By using feature points that are easy to process images, it is possible to recognize a pixel corresponding to an object among several pixels of a camera image. This is equivalent to recognizing the position or angle of an object in a camera image.

After the first step is performed, it may be after the feature point data set is obtained and after the model-view matrix and the perspective-projection matrix are obtained. Of course, the inverse of the model-view matrix and the inverse of the perspective-projection matrix can be obtained. If the object is replaced with another object after the first step is performed, even if the object is placed in a different position, whether the object is placed within the field of view of the camera, and if placed in the field of view, the position or angle of the object can be determined. . This can be obtained by determining which feature point has a difference in the image of the newly placed object based on the model-view matrix and the perspective-projection matrix obtained by the preprocessing process in the first step. For a new object, a new model-view matrix and a perspective-projection matrix corresponding to the object can be obtained in the second step, the object tracking process. This can be said to be equivalent to automatically recognizing the pose of the object including the position or angle of the new object.

The assembly position of the part for the object is not input into the camera or the augmented reality app, so it can be obtained by the user's teaching. When the user touches the assembly position of the part in the object image of the camera, touch coordinates may be generated. The augmented reality app can calculate the target coordinates of the assembly position of the part by using the inverse of the model-view matrix and the inverse of the perspective-projection matrix.

The obtained target coordinates are displayed on the screen and can be checked by the user, and are recorded in an augmented reality app to set the assembly position of the part or determine whether to assemble in place. On the other hand, the component assembly position setting process can be performed only once in the initialization routine.

Next, a process of generating augmented reality content will be described.

Referring to FIG. 4, the augmented reality content includes a 3D box indicating a component assembly position of each component or a component assembly sequence of each component, a coordinate text representing target coordinates or touch coordinates of the component assembly location, and a user assembly location or 3D It may include at least one of a part image or part information including part text that is popped up when the box is touched, an assembly status indicator that displays a component assembly state, and a voice message that informs the assembly state by voice.

In FIG. 4, the z-axis position of the 3D box may be set to 0. In order to immediately notify the operator when an assembly mistake occurs, a voice message through the smartphone microphone may be generated. Voice message generation can be implemented by utilizing the text to speech (TTS) function of a terminal or smartphone. Among augmented reality contents, a 3D box is the most important element, and a 3D box must be accurately displayed at a predetermined part assembly position of an object, and the 3D box creation process is as follows.

1) You can create a virtual camera to render a 3D box or display a 3D box on the terminal screen. Each 3D box may be a window displaying images input/output to each virtual camera.

2) The third step of FIG. 1, which is an object tracking step of recognizing an object from an object image (valve body image) displayed on the terminal screen after being photographed with a terminal camera, may be performed.

3) When the object is recognized in the object tracking step, the inverse model-view matrix, which is an inverse matrix of the model-view matrix, is calculated, and the position, rotation, and rotation of the terminal camera based on the object using the inverse model view matrix. A pose of the terminal camera including at least one of the directions may be obtained. That is, it is possible to obtain a model view matrix of the terminal camera including pose information between the object and the terminal camera.

4) The pose of the virtual camera can be set the same as that of the terminal camera. That is, a model view matrix of a virtual camera including pose information between an object and the virtual camera can be obtained.

5) Perform calibration to include at least one of the resolution, size, focal-length, field of view, and aspect ratio of the terminal camera. The desired terminal camera parameters may be obtained, and the perspective projection mode of the virtual camera or the perspective projection matrix of the virtual camera may be obtained using the terminal camera parameters.

6) After creating the geometry of the virtual camera including the model view matrix or the perspective projection matrix of the virtual camera, the 3D boxes can be placed at the same geometry location as the part assembly location found in the part assembly location setting process. .

Each 3D box may be a virtual camera that is a means for real-time detection of whether parts are assembled according to a predetermined position and order. The assembly information identified by each 3D box can be viewed, and if an assembly error occurs, the operator can be immediately informed and corrected.

There may be a method for each 3D box or each virtual camera to use the feature points of the part or the assembly location to determine the good or bad of the part assembly.

In addition, additional features, such as shape or color, can be given to the part. This method is to easily check the presence or absence of detection of parts while being less affected by surrounding environmental noise. For example, a part of the part may be painted with a color that can be distinguished from the valve body and detected. Referring to FIG. 5, in the image, a part of the head of the component to be assembled may be painted red to show the assembly. If a red color is suddenly detected in the virtual camera or 3D box, it can be determined that the part is well assembled. Although a separate process of coloring parts is required in advance, it is possible to very accurately and quickly determine whether parts are assembled through a simple algorithm to determine the color.

6 to 9 are views showing a result of assembly of a component to a valve body as an object. Referring to FIGS. 6 to 9, five components may be assembled in order at predetermined positions of the valve body 10.

The number displayed in the 3D box displayed at the assembly location of the part indicates the assembly order, and the currently assembled part can indicate the color of the 3D box in yellow. Where the current assembly is not in order, the user can be informed with a red 3D box.

The assembly status indicator is displayed.If the assembly process is normally performed, the assembly status is displayed as OK, and if an assembly error occurs, it is displayed as ERR so that the operator can process it after confirmation, and when the work is completed normally, END may be displayed. have.

If a part assembly location or 3D box is touched on the smartphone screen during assembly, part information including an image or part text of the part to be assembled at that location is displayed at the bottom of the screen so that the operator can check it.

The step-by-step description with reference to FIGS. 6 to 9 is as follows.

Referring to FIG. 6, part 1 (part1) (for example, screws) is normally assembled, and it is time to assemble part 2 (part2) indicated by a yellow 3D box.

During work, if the operator touches the location where the part 2 will be assembled (inside the yellow 3D box) to check the part 2, a red dot is displayed inside the 3D box and contains the name (part2) or image of the part 2. Part information can be displayed at the bottom of the screen. OK can be displayed on the assembly status indicator.

Referring to FIG. 7, it is time to assemble part 1 (part1), part 2 (part2), and part 3 (part3) normally, and then to assemble part 4 (part4) marked with a yellow 3D box. OK may be displayed on the status indicator.

Referring to FIG. 8, the first part (part1) and the second part (part2) are normally assembled, and the turn to assemble part 3 (part3), which is currently displayed as a yellow 3D box, or part 4 (part 4). A mistake may occur due to assembly, and an ERR-box may be displayed on the assembly status indicator for the location of the 4th part.

A voice message (for example, “a mistake has occurred”) is delivered to the operator through the smartphone's microphone, and the ERR may be displayed on the assembly status indicator.

Referring to FIG. 9, five parts assembly work is completed, and END may be displayed on the assembly status indicator.

In the present invention, since the assembly condition of all the parts must be detected within the time when the operator assembles each part, the period of the image processing detection routine can be set in consideration of the assembly time of the operator.

10: object (valve body)
S1: first step S2: second step
S3: third step S4: fourth step
S5: the fifth step S6: the sixth step
S7: the seventh step S8: the eighth step
S9: 9th step

Claims (6)

Extracting feature point data of the object;
Driving an augmented reality app;
Tracking the object;
Setting a component assembly position after tracking the object;
Detecting whether a component is assembled;
Projecting augmented reality content onto the object image;
Notifying the operator when an assembly mistake is confirmed;
Including,
After the terminal camera is installed at the position where the object is visible and the augmented reality app is driven, the pose of the object is calculated based on the feature point data of the object,
The object tracking step of finding each pose for each newly introduced object is repeatedly performed,
After the object tracking is completed, when a part assembly position is touched in the object image displayed on the terminal screen, touch coordinates of each part assembly position are generated,
The augmented reality app extracts the touch coordinates (POIc) from the terminal screen, then obtains the inverse matrix of the model-view matrix and the perspective-projection matrix, and uses the inverse matrices to obtain target coordinates (POIs) corresponding to the touch coordinates. ),
Creating a virtual camera for displaying the 3D box as the augmented reality content on the terminal screen,
Performing the object tracking step of recognizing the object from the object image displayed on the terminal screen,
When the object is recognized in the object tracking step, a pose of the terminal camera including at least one of a position, rotation, and direction of the terminal camera based on the object is calculated,
Set the pose of the virtual camera to be the same as that of the terminal camera,
An object inspection method for displaying the 3D box at the same position as the part assembly position calculated in the part assembly position setting process.
The method of claim 1,
In the process of passing from the step of tracking the object to the step of setting the part assembly position, if the augmented reality app has not determined the part assembly position setting, an object to newly set the part assembly position of the object method of inspection.
delete The method of claim 1,
The augmented reality content is a 3D box indicating the assembly position of each part or the assembly order of each part, a coordinate text indicating target coordinates or touch coordinates of the part assembly location, and a pop-up when a user touches the part assembly location or 3D box An object inspection method comprising at least one of a component information including an image of a component or text of a component, an assembly state indicator displaying a component assembly state, and a voice message indicating the assembly state by voice.
delete The method of claim 1,
Each 3D box is displayed on the terminal screen for each part assembly position of the object image,
Each of the 3D boxes is displayed in a color that is distinguished by the assembly order of each part,
When each of the 3D boxes is touched, part information of the corresponding part is displayed on the terminal screen.

Images