KR101535801B1 - Process inspection device, method and system for assembling process in product manufacturing using depth map sensors - Google Patents

Abstract

TECHNICAL FIELD [0001] The present invention relates to a technique for detecting an error in an assembling process that occurs when a worker assembles parts in a manufacturing industry or the like.
A depth map sensor for generating position information at a plurality of predetermined tracking points corresponding to an assembler for assembling at least one object to be assembled; A camera for generating image information at a target position on which the assembly object is to be placed based on position information generated from the depth map sensor; And an inspection apparatus for controlling operations of the depth map sensor and the camera, and determining whether the objects to be assembled are assembled in a predetermined order based on the position information and the image information.

Description

TECHNICAL FIELD [0001] The present invention relates to an apparatus for inspecting an assembly process in an assembly process using a depth map sensor,

TECHNICAL FIELD [0001] The present invention relates to a technique for detecting an error in an assembling process that occurs when a worker assembles parts in a manufacturing industry or the like.

Even in recent years, in the manufacturing field, a method of manually assembling a large number of parts by a worker to manufacture a finished product has been frequently performed.

However, when the parts are manually assembled, defects may occur as errors occur in the assembling sequence.

In particular, assembly order errors for products requiring large-scale components, such as automobiles, lead to defects in the final product, thus requiring a huge error checking cost.

SUMMARY OF THE INVENTION In view of the foregoing, it is an object of the present invention to provide an apparatus, method and system for inspecting an assembling process, which enables an operator to automatically detect a mistake or an error in a process of assembling a plurality of parts by hand .

In order to achieve the above object, in one aspect, the present invention provides a depth map sensor for generating position information at a plurality of predetermined tracking points corresponding to an assembler for assembling at least one object to be assembled; A camera for generating image information at a target position on which the assembly object is to be placed based on position information generated from the depth map sensor; And an inspection apparatus for controlling operations of the depth map sensor and the camera, and determining whether the objects to be assembled are assembled in a predetermined order based on the position information and the image information.

According to another aspect of the present invention, there is provided an image forming apparatus including: an assembling order setting unit that sets an assembling order for at least one object to be assembled to complete one object; A position determining unit for determining a position of each of a plurality of predetermined tracking points corresponding to an assembler that assembles the object to be assembled based on the position information received from the depth map sensor; And an assembly order determination unit for determining whether each of the assembly objects is assembled according to a predetermined order according to the image information received from the camera driven based on the determination result of the position determination unit.

According to another aspect of the present invention, there is provided a method of manufacturing an assembly, comprising: an assembly sequence setting step of previously setting an assembly sequence for at least one object to be assembled to complete one object; A step of determining a position of each of a plurality of predetermined tracking points corresponding to an assembler that assembles the object to be assembled based on the position information received from the depth map sensor; A camera driving step of determining whether the camera is driven according to a determination result in the tracking point position determination step; And an assembly order determination step of determining whether each of the objects to be assembled is assembled in a predetermined order based on the received image information and receiving the image information from the camera driven according to the determination in the camera driving step do.

As described above, according to the present invention, it is possible to automatically detect an error of an assembly in an assembling step before error checking is performed on a final assembly, thereby reducing the time and cost burden in the manufacturing process .

In addition, when an error occurs in the assembling step, the operator can be notified immediately, thereby improving the assembling accuracy of the final assembly.

FIG. 1 is a configuration diagram showing a configuration of an assembly process inspection system according to an embodiment of the present invention.
2 is an exemplary view showing a side structure of the assembly process inspection system of FIG.
FIG. 3 is a block diagram illustrating the internal structure of an apparatus for inspecting an assembly process according to an embodiment of the present invention.
4 is a flowchart illustrating a method of inspecting an assembly process according to an embodiment of the present invention.
5 is an exemplary diagram showing a setting interface for setting an assembly order in an embodiment of the present invention.
6 is an exemplary diagram illustrating an example of recognizing an operation of a worker based on a positional relationship between respective tracking points in an embodiment of the present invention.
FIG. 7 is a diagram illustrating an example of an image in a normal assembled state and an image in an erroneous state according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

FIG. 1 is a configuration diagram showing a configuration of an assembly process inspection system according to an embodiment of the present invention, and FIG. 2 is an exemplary view showing a side configuration of the assembly process inspection system of FIG.

Referring to FIG. 1, an assembly process inspection system according to an embodiment of the present invention includes a sensor 20 for generating position information for recognizing movement of an assembler H, And an inspection device 30 for determining whether or not each of the objects to be assembled is assembled in accordance with the assembling order based on the information received from the sensor 20 and the camera 10 .

In addition, before the assembly is performed, the first assembly table T1 may include a plurality of assembly objects, and the second assembly table T2 may be configured to move the assembly objects. The first assembly table T1 and the second assembly table T2 are used to distinguish positions where the objects to be assembled are placed before and after the assembly. ) May be distinguished from each other.

The camera 10 can capture an image of the second assembly table T2, and can capture images of the first assembly table. The camera 10 may be a general RGB camera such as a CCD camera, but may be a vision sensor camera.

The camera 10 may include a first camera for generating image information on the first assembly table T1 side and a second camera for generating image information on the second assembly table T2 side, One camera with a plurality of lenses may be applied.

The sensor 20 is an element provided for recognizing the movement of the worker H and includes a plurality of tracking points preset to correspond to the worker's body part including the hand of the worker H The position information can be generated by detecting the position of the position information in real time.

Preferably, the sensor is a 3D depth map sensor.

FIG. 2 is a view for explaining the positional relationship between the camera and the sensor. Referring to FIG. 2, in the assembly process inspection system according to the embodiment of the present invention, And the sensor 20 may be fixedly arranged behind the camera 10 and may be inclined at a predetermined angle so as to recognize the worker H in front of the work table .

FIG. 3 is a block diagram illustrating the internal structure of an apparatus for inspecting an assembly process according to an embodiment of the present invention.

3, the apparatus for inspecting an assembly process according to an exemplary embodiment of the present invention includes an assembly order setting unit 110, a position determination unit 120, an assembly order determination unit 140, and a controller 160 do.

In addition, it may further include an image determination unit 130 and an error notification unit 150.

The assembly order setting unit 110 sets the assembly order in advance for at least one or more objects to be assembled to complete one assembly. The assembly order setting unit 110 reads the assembly sequence information stored in advance for the selected assembly and displays the assembled assembly information to be inspected It can be set as assembly order for water.

In addition, the assembly order setting unit 110 may set the input information as an assembly order when each assembly object and the order for completing the assembly are input from the user. In this case, the assembly order setting unit 110 Can provide the user with a setting interface for receiving the respective objects to be assembled and their order from the user.

5 is an exemplary diagram showing a setting interface for setting an assembly order in an embodiment of the present invention.

For example, if the initial positions of the n pieces of assembly objects, that is, the respective positions on which the n pieces of assembly objects are placed in the first work platform are {P1s, P2s, P3s, ..., Pns} P1t, P2t, P3t, ..., Pnt}, which is the target position to be placed in. In other words, the first part P1s must be in the P1t position and the P2s part should be in the P2t position.

The positions of these two vectors can be obtained by using the measured distance calculated using the depth map sensor or by changing the actual coordinate data received from the sensor to the window coordinate system projected on the window.

As shown in FIG. 5, the setting interface UI includes first and second input areas A and B for setting an initial position and a target position, And a plurality of function buttons for inputting a command for further inputting a position or a setting completion command when the input is completed. Therefore, the user can set the position value on which the assembly object is placed to the initial position, the position value to be placed by the assembly object by assembly, as the target position, and can sequentially input values in accordance with the assembly order of the respective assembly objects.

For example, when the objects to be assembled of X, Y, and Z are to be assembled one by one, the user sets the initial position and the target position for the X-assembly object at the initial position 1 and the target position 1 through the setting interface UI The initial position and the target position for the Y-assembly object can be set to the initial position 2 and the target position 2, respectively. By setting the initial position and the target position for the Z-assembly object to the initial position 3 and the target position 3, Can be set.

If the difference between the position value and the position information detected from the sensor is within the threshold value, the object to be assembled is present in the initial position or the target It can be determined that the position has been reached.

The position determination unit 120 determines a position of each of a plurality of predetermined tracking points corresponding to an assembler H for assembling the object to be assembled based on the position information received from the depth map sensor 20. [

The depth map sensor 20 can detect positions of a plurality of tracking points such as an operator's hand and fifteen joints in real time to generate position information.

The depth image refers to the distance between the objects on the space where the sensor 20 is placed as one image. The method of generating the image may be a time-on-flight (TOF), a structured light, Camera and the like. For example, in the case of Kinect, it consists of a part for projecting infrared light using an infrared light (IR) projector and a camera for accepting and analyzing the infrared light. An IR projector projects an image of a specific pattern onto a space, analyzes the image through a camera that receives the image, measures the distance between the object and the sensor, and rasterizes the image to generate an image. Sensors such as Kinect can use the function to obtain the position, joint position and orientation value of the user by analyzing the depth image through middleware such as OpenNI (Open Natural Interaction), which is an open source project. The position and orientation values of the open joint can be obtained in real time.

The position determination unit 120 may track the positions of the respective tracking points in real time based on the position information received from the depth map sensor 20 to determine whether the assembler reaches the target position by holding the object to be assembled, , And judges whether there is an interaction between the assembler and the object to be assembled. The position determiner 120 may use various heuristic functions to recognize the gesture. A more detailed description of the heuristic function will be described later with reference to the accompanying drawings.

Here, the interaction between the co-ordinator and the object to be assembled refers to a movement such as holding the object with the hand and turning it, or grasping the object to be assembled and grasping the abnormality.

The position determination unit 120 may track the movement of the co-ordinator based on the received position information. The position determination unit 120 may correspond to a part of the co-operative body used for assembling the head and the hand based on the tracking point corresponding to the waist of the co- The change of the position value can be calculated or the directionality can be determined for the set tracking point.

The assembling sequence determining unit 140 determines whether each of the assembling objects is assembled according to a predetermined order according to the image information received from the camera driven based on the determination result of the position determining unit 120. [

When it is determined that the object to be assembled has reached the preset target position as a result of the determination by the position determination unit 120, the control unit 160 controls the operation of the camera 10 And controls the camera 10 to receive image information according to the driving of the camera 10. [

If it is determined that an error has occurred in the assembly sequence as a result of the determination of the assembly sequence determination unit 140, the error notification unit 150 may be configured to display the message in the form of a message An alarm may be outputted in a form of sound through a speaker or the like, or may be implemented in various forms such that the assembler can recognize an error such as a vibration.

Meanwhile, the inspection apparatus according to an embodiment of the present invention may further include an image determination unit 130 that identifies the object to be assembled from the received image information. If the object to be assembled identified from the image information is preset If it is determined that the object to be assembled is not an object to be assembled, the control unit 160 may generate a control command to inform the error notification unit 150 of the error in the assembly procedure.

Accordingly, even if an object to be assembled that does not match the set assembly order is caught by the operator, the operator can recognize that the object is erroneously held, thereby improving the accuracy of assembly.

4 is a flowchart illustrating a method of inspecting an assembly process according to an embodiment of the present invention.

Referring to FIG. 4, the assembly process of the assembly process inspection apparatus according to an embodiment of the present invention The inspection method includes an assembly order setting step S120, a tracking point position determination step S140, a camera driving step 150, and an assembly order determination step 160. [

In the assembly order setting step (S120), the assembly order is previously set and stored for at least one or more objects to be assembled to complete one assembly.

Next, based on the position information received from the depth map sensor, a position is determined for each of a plurality of predetermined tracking points corresponding to the cooperators assembling the object to be assembled (S130)

And determines whether the camera is driven according to the determination result of the tracking point position determination step. That is, if it is determined that the position of the tracking point corresponding to the operator's hand has reached the predetermined target position with respect to the object to be assembled, the camera driving control signal may be generated to be transmitted to the camera (S140, S150)

Next, the image information is received from the driven camera, and it is checked whether each of the objects to be assembled corresponds to the assembling order set in the assembling order setting step based on the received image information (S160)

If it is determined that there is an error in the assembling sequence, the operator informs the operator of the error.

Here, the inspection method according to an embodiment of the present invention may further include a pre-processing step (S110) such as zero point correction or tracking point position correction for the depth map sensor and the camera before the assembling order setting step.

Each step will be described in more detail as follows.

The preprocessing step (S110) is a step of correcting the zero point correction or the tracking point position for the depth map sensor and the camera before the assembly order setting step. The 3D depth sensor and the camera are physically installed, You can use the SDK middleware to perform the calibration. The physical sensor and the position can be set as shown in Figs. 1 and 2 according to the size of the assembly process product (assembly), and the position and size of the table on which the assembly is performed.

The three-dimensional sensor is preferably installed at a distance of 3 meters or more from the co-ordinator, and the RGB camera is installed near the position of the coarse object to image the coarse coarse manipulating process with respect to the coarse object with high resolution, Device.

After the setting of the sensor and the camera is completed and the positioning of the assembly sequence is completed, the three-dimensional joint positions of the fifteen joints are tracked in real time using the middleware such as the manufacturer's SDK (S130) The depth map image can be obtained and analyzed by middleware to provide 3-dimensional position information for 15 joints in front of the sensor. The information about the fifteen joints transmitted through the middleware includes the orientation information of the location. The position of the waist joint can be used to estimate the overall position of the co-ordinator, and the position and orientation of the hand, which is directly used for assembly, is used as sensing data for grasping the interaction with the co-ordinate object. The position and direction data of both hands coming in real time are used as inputs of various heuristic functions and used to recognize the gesture of the present cooperator.

6 is an exemplary diagram illustrating an example of recognizing an operation of a worker based on a positional relationship between respective tracking points in an embodiment of the present invention.

6, the position determination unit 120 calculates the distance between the tracking points P2a and P2b corresponding to the hands of the operator and the tracking point P1 corresponding to the head, If the value is greater than or equal to a preset value, it can be determined that the operator is taking an action of raising the hand near the head, and the operation can determine that the part is correct and the gesture is an eye glare.

Similarly, the position determination unit calculates the distance between the tracking points P2a and P2b corresponding to the hands of the operator, calculates the angle, and determines whether the calculated distance value is less than a preset value and the angle between the tracking points is within a preset range It can be determined that both hands are in an operation close to each other, and the operation can be judged to be a gesture for confirming whether or not the worker can operate the parts for checking the clearance inside the parts.

However, the task of automatically detecting the assembling sequence of the assembler to perform the assembling operation is to check whether the error is caused by the real time tracking technology of the three-dimensional movement of the assembler, the movement trace of the hand along with the movement of the whole worker, Techniques for confirming whether or not they are assembled in the correct order are required.

Since the assembly process in the manufacturing industry used in the present invention continues the operation of touching the assembly object mainly by using the hand, it is not possible to grasp whether the assembly is assembled in the correct order only by locating the hand provided by the middleware. For this purpose, a camera is installed to confirm whether or not the operator picks up the object to be assembled by using his / her hand. The camera continues to capture images of the specific location of the assembly where the assembly actually takes place and continually grasps whether the assembly is in the wrong order.

In this case, if capturing and analyzing the image before the assembly object is transferred to the assembly stage through the hand may have a negative effect on the overall system performance, the joint position tracking and the determination of the interaction between the assembly and the assembler When the distance between the position and the assembly position of the assembly stand falls below the threshold value, it is preferable to start the RGB camera and image analysis (S140)

Whether the object to be assembled has been moved to the target position is determined by checking whether the object and the object are within a predetermined distance to the target position set in the assembly order setting step and the object is continuously placed at the position for a predetermined time. That is, if the current position of the hand is p and the target position is t, if the distance between the two falls within a predetermined threshold value d, the camera is automatically driven through the camera driving step (S150)

Whether the camera is continuously positioned at the target position searched through the driven camera can be recognized through the 2D pattern matching method. Camera assembly sequence errors can be identified using a template - based 2D image - based pattern matching algorithm. That is, after obtaining the image Ic when the assembling sequence is correctly assembled and the image Ic 'when the assembling sequence is not correct, as the reference image, the image currently coming in through the camera is close to any one of Ic and Ic' (S160)

FIG. 8 is a diagram illustrating an example of an image in a normally assembled state and an image in an erroneous state according to an embodiment of the present invention.

That is, as shown in FIG. 8, when FIG. 8A is stored as an image when assembled, it is possible to compare FIGS. 8B and 8A, which are image information received from the camera, If the video information does not match, it can be determined that there is an error.

The assembling order can be set to two cases. When all the assemblies are assembled while being stacked at one position, all target positions of each assembled object set in the setting interface are set to the same, and the image pattern matching method The assembly sequence error can be recognized.

When all the assemblies are assembled while being placed at different positions, all the target positions of the setting interface are set to be different from each other. After determining whether the position of the hand obtained through the 3D depth map sensor approaches the target position, And determines whether the object to be assembled is placed or not through the image pattern matching.

Further, such a process is continuously repeated to detect whether the objects at the original position are placed in the predetermined order in the target position. If it does not match the order set in the assembly sequence setting step, it informs the assembler in the notification step.

That is, when the position of the tracking point corresponding to the hand reaches the target position, the assembly of the assembly is checked to see if the position coincides with the position set in the assembly order setting step. If this sequence is correct, a success message can be generated and output, otherwise an error message can be output to the monitor along with the sound and delivered to the assembler.

Although the method of inspecting the assembling process according to the embodiment of the present invention has been described above with reference to FIGS. 4 to 7, it should be understood that the present invention is not limited to the above- Depending on the implementation method, the procedure of each step may be changed, or two or more steps may be integrated, or one step may be performed in two or more steps.

In addition, the program implementing the assembling process inspection method according to the embodiment of the present invention can be executed by the computer by being recorded on a recording medium which can be read by a computer and executed by a computer.

As described above, in order for the computer to read the program recorded on the recording medium and execute the assembly process inspection method implemented by the program, the above-mentioned program may be stored in a storage medium such as C, C ++, JAVA, And may include a code encoded in a computer language.

The code may include a function code related to a function or the like that defines the functions described above and may include an execution procedure related control code necessary for the processor of the computer to execute the functions described above according to a predetermined procedure.

In addition, such code may further include memory reference related code as to what additional information or media needed to cause the processor of the computer to execute the aforementioned functions should be referenced at any location (address) of the internal or external memory of the computer .

In addition, when a processor of a computer needs to communicate with any other computer or server, etc., to perform the above-described functions, the code may be stored in a computer's communication module (e.g., a wired and / ) May be used to further include communication related codes such as how to communicate with any other computer or server in the remote, and what information or media should be transmitted or received during communication.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. The codes and code segments constituting the computer program may be easily deduced by those skilled in the art. Such a computer program can be stored in a computer-readable storage medium, readable and executed by a computer, thereby realizing an embodiment of the present invention. As a storage medium of the computer program, a magnetic recording medium, an optical recording medium, or the like can be included.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

Claims (8)

A depth map sensor for generating position information at a plurality of predetermined tracking points corresponding to an assembler for assembling at least one object to be assembled;
A camera for generating image information at a target position on which the assembly object is to be placed based on position information generated from the depth map sensor; And
An inspection device for controlling the operation of the depth map sensor and the camera and for determining whether the objects to be assembled are assembled in a predetermined order based on the position information and the image information,
A system for inspecting an assembly process using a depth map sensor. In an assembly process inspection apparatus,
An assembly order setting unit that sets an assembly order for at least one or more objects to be assembled to complete one assembly;
A position determining unit for determining a position of each of a plurality of predetermined tracking points corresponding to an assembler that assembles the object to be assembled based on the position information received from the depth map sensor;
An assembling sequence determining unit for determining whether each of the assembled objects is assembled according to a predetermined order according to the image information received from the camera driven on the basis of the determination result of the position determining unit,
And a depth map sensor including the depth map sensor. 3. The method of claim 2,
The position determining unit may determine,
And determining whether each of the assembled objects moves from a predetermined start position to a target position by deriving a position value and a direction value at each tracking point based on the position information received from the sensor. Process inspection equipment. 3. The method of claim 2,
If it is determined that an error has occurred in the assembling sequence as a result of the assembling sequence determining unit,
Wherein the depth map sensor further includes a depth map sensor. 3. The method of claim 2,
A controller for generating and delivering a drive control signal to the camera when the position determination unit determines that the object to be assembled reaches a predetermined target position and transmitting the generated drive control signal to the camera,
And a depth map sensor including the depth map sensor. 3. The method of claim 2,
The assembly sequence determination unit may determine,
And comparing the image information of the normal assembled object of the pre-stored assembly object with the image information received from the camera, and determines that an error has occurred in the assembling sequence if the object image does not match. 3. The method of claim 2,
An image determination unit for identifying the object to be assembled from the received image information,
Wherein the depth map sensor further includes a depth map sensor. An assembling order setting step of previously setting an assembling sequence for at least one or more objects to be assembled to complete one object;
A step of determining a position of each of a plurality of predetermined tracking points corresponding to an assembler that assembles the object to be assembled based on the position information received from the depth map sensor;
A camera driving step of determining whether the camera is driven according to a determination result in the tracking point position determination step; And
An assembly order determination step of determining whether each of the objects to be assembled is assembled in a predetermined order based on the received image information and receiving image information from a camera driven in accordance with the determination in the camera driving step
Wherein the depth map sensor includes a depth map sensor.

Images