KR102647402B1 - Components fabrication scheduling apparatus - Google Patents

Abstract

The parts assembly planning device of the present invention includes an assembly instruction recognition module that recognizes assembly instructions; an assembly plan verification module that establishes and verifies an assembly plan using information recognized by the assembly instruction recognition module; And it may include an assembly plan visualization module that can provide the assembly plan as visual information.

Description

Components fabrication scheduling apparatus}

The present invention relates to a parts assembly planning device. In detail, it relates to a parts assembly planning device that recognizes static information describing the assembly of parts and establishes a part assembly plan.

A variety of robots can be applied to the production process to assemble parts. The process of changing a robot's work items consumes a lot of cost in robot hardware and software, and automatic parts assembly by a robot requires a separate parts assembly plan in addition to an instruction manual for humans.

With current technology, most robots for parts assembly work are responsible for part of the part assembly. Additionally, significant additional costs are incurred when modifying the design or adding work types.

As a related prior art, application number: 10-2013-0116516, 'Matrix-based component assembly order extraction method and device' has been proposed. This technology implements the connection relationship of the parts that make up the assembly model as a hierarchical connection graph between nodes, and the layer is created by extending the nodes based on the base portion.

This technology can be said to be a limited technique as it only provides an assembly sequence when the parts that make up the assembly model are provided.

Application number 10-2013-0116516 Matrix-based component assembly order extraction method and device

Against the background mentioned above, the present invention provides a parts assembly planning device that provides a part assembly plan that allows robots to automatically perform all parts assembly processes.

The present invention provides a parts assembly planning device that automatically generates an assembly plan that a robot can understand without human assistance.

The parts assembly planning device of the present invention includes an assembly instruction recognition module that recognizes assembly instructions; an assembly plan verification module that establishes and verifies an assembly plan using information recognized by the assembly instruction recognition module; And it may include an assembly plan visualization module that can provide the assembly plan as visual information.

The assembly instruction recognition module includes an assembly recognition unit that recognizes structural parts to be assembled from the assembly instructions; A structural part type recognition unit that recognizes the type of the recognized structural part; a character recognition unit that recognizes characters presented in the assembly instructions and recognizes at least fastening parts through the characters; A fastening part behavior recognition unit that recognizes the behavior of the fastening part; a coupling estimation unit that estimates a coupling position of the fastening part and the structural part using the structural part, the fastening part, and the behavior of the fastening part; And it may include a coupling position inference unit that infers a three-dimensional coupling position of the structural part to which the fastening part is coupled.

The assembly instructions may be provided as two-dimensional information.

Three-dimensional information may be provided to the three-dimensional combined location inference unit.

The assembly recognition unit includes an assembly area detection unit that detects an assembly area including the structure product; and a structural part recognition unit that divides the assembly area and detects the area of each structural part in order to recognize the structural part within the assembly area.

The structural part recognition unit may recognize the structural part as an image.

The structural part type recognition unit is provided with a third learning model, and the third learning model includes: a three-dimensional feature extraction unit configured with a 3D convolutional network to extract features of the three-dimensional model of the structural part; a two-dimensional feature extraction unit composed of a 2D convolutional network to extract features from the two-dimensional image of the three-dimensional model; and a learning department may be included.

The learning unit learns the features extracted from the three-dimensional feature extraction unit for one structural part to be similar in distance to the features extracted from the two-dimensional feature extracting unit for the same structural part; And the features extracted from the three-dimensional feature extraction unit for one structural part may be learned to be distant from the features extracted from the two-dimensional feature extraction unit for another structural part.

The text recognition unit includes a text area recognition unit that recognizes a text area containing a text; and a character inference unit that infers a character from the character area.

The assembly permutation can be determined by the character recognition unit.

The coupling estimation unit includes an axial direction detection unit that detects the coupling axis direction of the fastening part; And it may include a coupling position recognition unit that extends the coupling axis direction and recognizes a point where the extension line meets the structural part.

The coupling estimation unit may provide the coupling position for a two-dimensional model of the structural part.

The coupling position inference unit may adjust the 6DoF posture of the three-dimensional model of the structural component to obtain the 6DoF posture that matches the two-dimensional model of the structural component.

The coupling position inference unit may infer the coupling position on a three-dimensional model of the structural part.

According to the present invention, assembly instructions that humans can understand can be provided as part assembly plans that robots can understand.

According to the present invention, it is possible to provide a parts assembly plan that allows all parts assembly processes to be performed by a robot.

Various other effects of the present invention can be understood through the detailed description of the present invention.

1 is a schematic configuration diagram of a parts assembly planning device according to an embodiment.
Figure 2 is a diagram showing the detailed configuration of the assembly instruction recognition module of Figure 1.
Figure 3 is a view showing the assembly area and the structural component area in the assembly instructions.
Figure 4 is a diagram showing the learning steps of the character area.
Figure 5 is a diagram showing the learning stage of character recognition.
Figure 6 is a diagram explaining the operation of the structural part type recognition unit.
Figure 7 is a diagram showing the operation results of the fastening part behavior recognition unit.
Figure 8 is a diagram illustrating the direction of the coupling axis detected in the fastening part and the point where the coupling axis meets the structural part.
Figure 9 is a diagram showing the process of matching the two-dimensional image output from the differentiable renderer and the two-dimensional image output from the combination estimation unit.
Figure 10 is a diagram illustrating 6DoF of a three-dimensional model.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the following embodiments, and those skilled in the art who understand the spirit of the present invention can easily make other embodiments included within the scope of the same spirit by adding, changing, deleting, and adding components, etc. However, this may also be included within the scope of the present invention.

In the description of the drawings, identical or similar components are assigned the same reference numbers regardless of reference numerals, and overlapping descriptions thereof may be omitted.

The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves.

In describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted.

The attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes, equivalents, and changes included in the spirit and technical scope of the present invention are not limited. It should be understood to include water or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Singular expressions may include plural expressions, unless the context clearly indicates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

1 is a schematic configuration diagram of a parts assembly planning device according to an embodiment.

Referring to FIG. 1, the parts assembly planning device of the embodiment includes an assembly instruction recognition module (1) that recognizes assembly instructions, and an assembly plan that is established and verified using the assembly instructions recognized by the assembly instruction recognition module (1). It may include a plan verification module (2) and an assembly plan visualization module (3).

The assembly instructions may refer to data that can be recognized by humans. The assembly instructions may refer to two-dimensional data explaining assembly. The assembly instructions may refer to a booklet or information that has been digitized from the booklet.

The assembly instruction recognition module 1 can recognize assembly instructions and understand the fastening structure of each component. The assembly instruction recognition module 1 can input two-dimensional assembly instructions and learning models. The learning model may be a variety of learning models in which various information required for recognition of the assembly instructions is learned in advance as needed. In the learning model, a three-dimensional model in which each part of the assembly instructions is modeled in three dimensions can be learned. The three-dimensional model may be processed into a number of virtual data before being learned. The processed data may be a variety of information created through a process in which a three-dimensional model is placed in three-dimensional space and cameras, light sources, and rendering variables are adjusted. Here, various information may include similar images, instance masks, surface normals, and depth information images.

The assembly plan verification module 2 can establish an assembly plan. The assembly plan verification module 2 can establish multiple provisional assembly plans. The provisional assembly plan may go through a verification step. The provisional assembly plan can be verified by ensuring that the joining positions between parts are well mapped in two-dimensional and three-dimensional data. For example, only a provisional assembly plan in which the joining positions of two-dimensional parts presented in the assembly instructions are well mapped to the joining positions of three-dimensional parts can be output as an assembly plan. For example, a provisional assembly plan that best maps the joining positions of two-dimensional parts presented in the assembly instructions to the joining positions of three-dimensional parts can be output as a verified assembly plan. Here, the three-dimensional part may be a part that has been pre-modeled in three dimensions that the robot can recognize.

The assembly plan visualization module 3 can visualize the assembly plan using various digital tools. You can finally check the part assembly plan through the visualized assembly plan. By visualizing the assembly plan, an electronic file that can be input by a robot can be created. Visualization tools can be Blender or Python.

FIG. 2 is a diagram showing the detailed configuration of the assembly instruction recognition module of FIG. 1. Referring to FIG. 2, the configuration and operation of the assembly instruction recognition module will be briefly explained.

The assembly manual recognition module includes an assembly recognition unit 10 that recognizes each structural part assembled from the assembly manual, a structural part type recognition unit 30 that recognizes the type of structural part recognized in the assembly recognition unit, and an assembly manual. A character recognition unit 20 that recognizes characters with meanings such as numbers and letters presented in, a fastening part behavior recognition unit 40 that recognizes the behavior of fastening parts, the structural parts, the fastening parts, and the fastening. A coupling estimation unit 50 that estimates the coupling of structural parts and fastening parts in two dimensions by referring to the behavior of the parts and other information, and a three-dimensional coupling estimation unit 50 that infers the three-dimensional coupling positions of the structural parts and fastening parts and provides recognized assembly instructions. It may include a combination position inference unit 60.

Taking the assembly instructions for a chair as an example, the structural parts may include a plurality of pillars used in the chair, and the fastening parts may refer to fastening members such as screws that fasten the pillars. The character recognition unit 20 can recognize the fastening parts, their number, etc. The characters are not limited to numbers and letters and may include any object that can have meaning.

The operation of the assembly manual recognition module will be briefly explained.

The assembly recognition unit 10 can recognize each structural part to be assembled. Here, recognition can be understood as recognizing the two-dimensional shape of the structural part. Each recognized structural part is input into the structural part type recognition unit 30 so that the type of the structural part can be recognized. For example, the recognized structural part can be accurately identified as the second pillar used for the back of a chair.

The character recognition unit 20 can recognize characters presented in the assembly instructions and recognize various information required for the assembly process. The information may include assembly order information corresponding to the number and type of the fastening parts and page number. Information recognized by the character recognition unit may be transmitted to the combination estimation unit 50 and the fastening part behavior recognition unit 40.

The fastening part behavior recognition unit 40 can recognize the fastening operation of the fastening part. For the fastening operation of the fastening part, for example, movements such as rotation of the fastening part can be recognized.

The combination estimation unit 50 can estimate the combination of each part using the structural parts, the fastening parts, and the behavior of the fastening parts. Here, each piece of information is a character recognition unit (20). It can be transmitted from the structural part type recognition unit 30 and the fastening part behavior recognition unit 40.

The coupling estimation between the parts of the coupling estimation unit 50 can be inferred based on the two-dimensional assembly instructions. The two-dimensional coupling estimation can be inferred in three dimensions from the three-dimensional coupling position inference unit 60 of the structural part.

The inferred three-dimensional combined estimate can be verified in the assembly plan verification module (2) and visualized and output in the assembly plan visualization module (3).

The configuration and operation of the assembly instruction recognition module 1 will be described in more detail for each configuration.

In the assembly instructions, various components such as speech bubbles, instruction symbols, numbers, and fastening parts may be present in addition to the structural parts that make up the structure of the product. Therefore, in order to accurately recognize structural parts, it is desirable that the area containing the structural parts be recognized first. For this purpose, the assembly recognition unit 10 includes an assembly area detection unit 11 that detects an assembly area containing the assembled structure product, and divides the assembly area in order to recognize each structure product as an image in the assembly area. Thus, a structural part recognition unit 12 that detects the area of the structural part may be included.

The first learning model may be used in the operation of the assembly recognition unit 10. The first learning model can be constructed using the Cascade-RCNN method to recognize the assembly area, and the learning model can be built using the Mask-RCNN method to recognize the area of each structural part. The learning model, etc. may be input to U-net and further improved to provide the first learning model.

Figure 3 is a diagram showing the assembly area and the structural component area in the assembly instructions. Referring to FIG. 3, at least one assembly area containing the structural part is first detected. Afterwards, it can be confirmed that each structural part area is detected in the assembly area.

The type of each structural part detected by the assembly recognition unit 10 can be recognized by the structural part type recognition unit 30.

In the above assembly instructions, the structural parts have various types and shapes, and in particular, intermediate assemblies made of individual parts cannot be learned, making it difficult to recognize the types. To improve this problem, the type of structural part can be recognized using a two-dimensional and three-dimensional matching method. In other words, the type of the structural part can be recognized by recognizing the structural part recognized through the assembly instructions, that is, the two-dimensional model recognized by the assembly recognition unit 10, and the structural part that is the most similar three-dimensional model. .

A third learning model may be used in the operation of the structural part type recognition unit 30. The third learning model includes a three-dimensional feature extraction unit composed of a 3D convolutional network to extract features of the structural part itself of the three-dimensional model, and a 2D convolutional network to extract features from the two-dimensional image of the three-dimensional model. It can be composed of a configured two-dimensional feature extraction unit. Here, the two-dimensional image may have the same dimension as the image of the structural part presented in the assembly description.

The third learning model may include a learning unit that learns as follows using the feature extraction unit. The features extracted from the three-dimensional feature extraction unit for one structural part can be learned to have a similar distance to the features extracted from the two-dimensional feature extraction unit for the same structural part. The features extracted from the three-dimensional feature extractor for one structural part can be learned to be distant from the features extracted from the two-dimensional feature extractor for another structural part. At this time, triplet loss can be used as the loss function for learning. Through the above-mentioned learning, it is possible to match the structural parts of the two-dimensional model, that is, the structural parts in the assembly instructions, with the structural parts of the three-dimensional model. Through this, the type of structural part in the assembly description can be recognized.

Figure 6 is a diagram explaining the operation of the structural part type recognition unit.

Referring to FIG. 6, it can be seen that the type of structural part is recognized by calculating the similarity between the image of the structural part detected in the assembly instructions and the part candidate group provided through the third learning model.

In the assembly instructions, the number of fastening parts and the numbers of various parts may be written in numeric form. In addition, characters for various parts may be written. The character recognition unit 20 can recognize characters.

The character recognition unit 20 may include a character area recognition unit 21 that recognizes an area containing a character, and a character inference unit 22 that infers a character from the recognized character area. Through this, characters can be recognized in two steps: first recognizing the area containing the character, and then recognizing the character in the recognized area.

For this purpose, the second learning model can be learned in two steps: a text area and character recognition within the text area. Figure 4 shows the learning stage of the character area, and Figure 5 shows the learning stage of character recognition.

Referring to FIG. 4, in order to detect the text area, assembly instruction data including thousands of bounding boxes can be collected and labeled. Using this, Faster R-CNN, which is mainly used for object area detection, can be trained. To improve the character area, the contour within the recognized character can be recognized, and then outliers can be removed through RANSAC based on the standard score of the size of the bounding box.

Referring to Figure 5, for character recognition, the TPS-ResNet-BiLSTM-Att model can be used. Model application can be done by using a text renderer to create an image for virtual letter recognition and transfer learning the model learned from a large data set.

By operating the character recognition unit 20, the type and number of the fastening parts can be recognized. Through the operation of the character recognition unit 20, the permutation of the assembly according to the order of the pages can be identified. In other words, the conclusion of the front page may precede the conclusion of the back page. Here, pages can be recognized by the number of each page.

The information on the fastening part recognized by the character recognition unit 20 allows the fastening part behavior recognition unit 40 to recognize the behavior of the fastening part. For example, motion information such as whether a fastening part rotates a screw to the right can be recognized.

A fourth learning model can be used for the operation of the fastener behavior recognition unit 40. The fourth learning model can use Cascade Mask R-CNN. Through this, it can operate robustly against speech balloons. The dataset for learning may be in the form of an image with labeled speech bubbles, such as a slide picture. Performance is improved by utilizing context-aware data augmentation during the learning process. The fastening tool and the arrow indicating the fastening direction can also be recognized by learning the same deep learning model.

Figure 7 is a diagram showing the operation results of the fastening part behavior recognition unit.

Referring to Figure 7, it can be confirmed that the type and quantity of fastening parts are recognized, and the assembly behavior of fastening tools and fastening parts is recognized.

After the types and positions of the structural parts and the fastening parts are recognized, it is possible to estimate what kind of coupling relationship each part has. The combination estimation unit 50 can estimate the combination of fastening parts and structural parts. For this operation, the coupling estimation unit 50 includes an axial direction detection unit 51 of the fastening part that detects the coupling axis direction of the fastening part, and extends the coupling axis direction of the detected fastening part, so that the extension line is connected to the structural part. It may include a coupling position recognition unit 52 that recognizes the meeting point. Through this, the position on the structural part where each fastening part is fastened can be recognized on a two-dimensional image.

Figure 8 is a diagram illustrating the direction of the coupling axis detected in the fastening part and the point where the coupling axis meets the structural part.

The fifth learning model can be used for the operation of the combination estimation unit 50.

The Mask R-CNN model can be used to learn the fifth learning model. ResNet 50 can be used as a classification model. Data can be trained using images of fasteners that render a three-dimensional model.

The estimated coupling relationship of the coupling estimation unit 50 is performed on a two-dimensional image. Combination relationships in two-dimensional images need to be inferred from coordinates in three-dimensional space. To this end, the three-dimensional coupling position inference unit 60 can infer the coupling relationship on structural parts and fastening parts that are three-dimensional models.

The three-dimensional combined position inference unit 60 can optimize and derive a shape similar to the two-dimensional model in the given assembly instructions by adjusting the 6DoF (Degrees of Freedom) posture of the three-dimensional model. Here, a differentiable renderer can be used in the optimization process. Given a three-dimensional model and its 6DoF posture, the differentiable renderer can output a two-dimensional image taken at that posture. Optimization can be performed in a direction where the output two-dimensional image matches the two-dimensional image input to the combination estimation unit 50.

Through this process, the coupling relationship can be inferred in three dimensions. In other words, after the two two-dimensional images are matched, the joining position of the fastening member can be inferred from the three-dimensional model. This is because the 6DoF information of the three-dimensional model is preserved.

Figure 9 shows the process of matching the two-dimensional image output from the differentiable renderer and the two-dimensional image output from the combination estimation unit. Figure 9 shows optimization performed using the L2 distance in pixel units. Figure 10 is a diagram illustrating 6DoF of a three-dimensional model.

Through the above process, the assembly instructions described as a two-dimensional model can be recognized as a three-dimensional model. The recognized three-dimensional joint estimation can be verified in the assembly plan verification module 2. Finally, the accuracy of the assembly plan can be confirmed through a visualization process in the assembly plan visualization module 3. Afterwards, it can be output as an electronic file that the robot can recognize using a general-purpose file such as STEP.

The electronic file can be used as data that allows any robot to perform the same assembly work on the same assembly item.

According to the present invention, even for new parts (for example, intermediate assemblies, etc.) that have not been learned, the type of structure can be recognized using a two-dimensional-three-dimensional matching algorithm based on shape similarity.

According to the present invention, it is possible to operate on objects for which 6DoF pose estimation has not been learned through a pose estimation algorithm that optimizes using differences between pixels.

According to the present invention, a parts assembly plan for a robot can be conveniently provided.

1: Assembly manual recognition module
2: Assembly plan verification module
3: Assembly plan visualization module
10: Assembly recognition unit
20: Character recognition unit
30: Structural part type recognition unit
40: Fastening part behavior recognition unit
50: Combined estimation unit of structural parts and fastening parts
60: Three-dimensional combined position inference unit

Claims (10)

Assembly instruction recognition module that recognizes assembly instructions;
an assembly plan verification module that establishes and verifies an assembly plan using information recognized by the assembly instruction recognition module; and
Includes an assembly plan visualization module capable of providing the assembly plan as visual information,
In the assembly manual recognition module,
an assembly recognition unit that recognizes a two-dimensional model of the structural part to be assembled from the assembly instructions;
a structural part type recognition unit that recognizes the type of the structural part by recognizing a three-dimensional model of the structural part with a similar shape to the two-dimensional model of the structural part recognized by the assembly recognition unit;
a character recognition unit that recognizes characters presented in the assembly instructions, recognizes at least fastening parts through the characters, and determines an assembly sequence according to the page order of the assembly instructions;
a fastening part behavior recognition unit that recognizes the behavior of the fastening part recognized by the character recognition unit, the fastening tool, and the fastening direction;
Using the structural part recognized by the structural part type recognition unit, the fastening part and assembly sequence recognized by the character recognition unit, the behavior of the fastening part recognized by the fastening part behavior recognition unit, the fastening tool, and the fastening direction a coupling estimation unit that estimates a two-dimensional coupling position of the fastening part and the structural part; and
It includes a coupling position inference unit that infers a three-dimensional coupling position of the structural part to which the fastening part is coupled,
The structural part type recognition unit is provided with a third learning model,
In the third learning model,
a three-dimensional feature extraction unit composed of a 3D convolutional network to extract features of the three-dimensional model of the structural part;
a two-dimensional feature extraction unit composed of a 2D convolutional network to extract features from a two-dimensional image of the three-dimensional model of the structural part; and
A learning unit that learns using the three-dimensional feature extraction unit and the two-dimensional feature extraction unit is included,
The learning unit learns the features extracted from the three-dimensional feature extraction unit for one structural part to have a similar distance to the features extracted from the two-dimensional feature extracting unit for the same structural part, and For this, the features extracted from the three-dimensional feature extraction unit are learned to be distant from the features extracted from the two-dimensional feature extraction unit of other structural parts,
The learning unit uses triplet loss as a loss function for learning,
The assembly recognition unit includes an assembly area detection unit that detects an assembly area including the structural part; and a structural part recognition unit that divides the assembly area and detects the area of each structural part in order to recognize the structural part within the assembly area,
The structural part recognition unit recognizes the structural part as an image,
Part assembly planning device. According to claim 1,
A parts assembly planning device in which the assembly instructions are provided as two-dimensional information, and the combination position inference unit is provided with three-dimensional information. delete delete delete According to claim 1,
In the character recognition unit,
A text area recognition unit that recognizes a text area containing characters; and
A parts assembly planning device including a character inference unit that infers characters from the character area. delete According to claim 1,
In the combined estimation unit,
An axial direction detection unit that detects the direction of the coupling axis of the fastening part; and
A parts assembly planning device comprising a coupling position recognition unit that extends the coupling axis direction and recognizes a point where the extension line meets the structural part. According to claim 1,
A parts assembly planning device wherein the coupling estimation unit provides the coupling position with respect to a two-dimensional model of the structural part. According to clause 9,
The coupling position inference unit adjusts the 6DoF posture of the three-dimensional model of the structural part, obtains the 6DoF posture matching the two-dimensional model of the structural part, and infers the coupling position on the three-dimensional model of the structural part. Device.

Images