KR102434419B1 - Precision 3D scanning device and method using multi-point markers and stereo vision - Google Patents

Abstract

The present invention relates to a precision 3D scanning apparatus and method using multi-point markers and stereo vision, and more particularly, to extract information on large workpieces, stereo vision with multi-points artificial markers attached to large workpieces. It relates to a precision 3D scanning apparatus and method using a multi-point marker and stereo vision for generating a 3D coordinate system by measuring with a camera and extracting data from a photographed image, wherein an artificial marker with N multi-points is attached A stereo photographing unit for photographing a large machine tool using stereo vision cameras; and a marker recognition unit for searching the positions of the M markers in the 2D image of the large machine tool photographed from the stereo photographing unit and displaying in 2D coordinates; and a 3D matching unit converting the recognition positions of the M marker points searched by the marker recognition unit into 3D (3-Dimension); and a 3D coordinate system representation unit representing the positions of the 3D marker points derived from the 3D matching unit in a 3D coordinate system.

Description

Precision 3D scanning device and method using multi-point markers and stereo vision

The present invention relates to a precision 3D scanning apparatus and method using multi-point markers and stereo vision, and more particularly, to extract information on large workpieces, by attaching artificial markers of multi-points to large workpieces, stereo It relates to a precision 3D scanning apparatus and method using a multi-point marker and stereo vision for generating a 3D coordinate system by surveying with a vision camera and extracting data from a photographed image.

In modern industry, computer numerical control (CNC) machines are the most important components. Based on the designed CAD model, a computer numerical control (CNC) machine handles milling, lathes, routers, welding machines, lasers and many more. In this process, the workpiece is arbitrarily loaded on the machine table or fixed at the desired position using a specially manufactured fixture. However, in the case of thin plate parts or carbon fiber reinforced plastic products, geometrical errors can easily occur, and the position of the workpiece may deviate from the ideal position or deformation may occur during processing. Because these issues affect the quality of the final product, the workpiece positioning process detects the shape and position of the actual workpiece. Here, the sensed geometric information is used to estimate the Euclidean transform and to calibrate the tool path.

A contact probe, usually attached to a 3-axis/5-axis computer numerical control (CNC) machine or industrial robot, uses a touch trigger mechanism to measure the 3D coordinates of the contact point between the probe and the workpiece. Although this type of scanner has a high level of accuracy and automation, the acquisition time and scope of work are limited by the physical movement of a computer numerical control (CNC) machine or robot.

A number of scanning techniques are used, in which multiple line scans or image slices are combined to measure large parts and reconstruct the three-dimensional shape of objects. Laser triangulation to generate surface line scans of an object as used for example in surface CAD-CAM reconstruction applications, and optical coherence tomography to generate image slice scans of objects for internal object 3D reconstruction applications. (OCT), magnetic resonance imaging (MRI), ultrasound image, radar image, and sonar image search method.

In all these applications, in order to accurately reconstruct the object in three-dimensional space, it is necessary to recombine several individual scan lines or image slices with accurate knowledge of the spatial relationship between each scan line or image slice.

In the laser triangulation method, in the process of scanning an object in three-dimensional space, a laser stripe is projected on the surface of the workpiece to generate a projection curve on the surface, the projected curve is captured by the camera, and the 3D profile of the curve is measured between the camera and the light source. It is calculated based on spatial relationships. However, the measurement accuracy of this scanner also depends on the reflective properties, and scanning may not work properly if the surface is smooth and transparent, and there are limitations to scanning in a dynamic situation with large machine tools. this exists

Korea Publication No. (KR) 2020-0142392 (2020.12.22)

The present invention is in view of such a problem, and the present invention provides a precision 3D scanning apparatus and method using a multi-point marker and stereo vision that can stably repeat the divided area of an industrial field machine with a multi-point marker-based point and quickly detect it. is to provide

According to embodiments of the present invention, a precision 3D scanning device using a multi-point marker and stereo vision is a stereo imaging unit for photographing a large machine tool attached with an artificial marker having N multi-points through stereo vision cameras. and a marker recognition unit that searches the positions of the M markers in the 2D image taken from the stereo photographing unit and represents the M markers in 2D coordinates; and N in each of the M markers searched from the marker recognition unit. a 3D matching unit that converts the recognition positions of marker points into 3D (3-Dimension); and a 3D coordinate system representation unit representing the positions of the NxM 3D marker points derived from the 3D matching unit in a 3D coordinate system.

According to embodiments of the present invention, the stereo photographing unit photographs the artificial markers of the multi-points of 2D (2-Dimension) attached to the surface of the large machine tool, and has a stereo vision function and two or more of the stereo vision cameras.

According to embodiments of the present invention, the stereo imaging unit repeatedly photographs the surface of the large machine tool divided by nxm from each of the two or more stereo vision cameras, and the stereo vision camera located in each divided area From both sides (right and left).

According to embodiments of the present invention, the marker recognition unit is a marker checking unit for recognizing the M number of 2D multi-point markers in the areas of the large machine tool surface obtained from the stereo imaging unit; and a marker point search unit that searches the positions of the two-dimensional artificial marker multi-points searched by the marker checking unit.

According to embodiments of the present invention, the marker recognition unit further includes a 2D coordinate representation unit indicating the positions of the 2D marker points searched by the marker point search unit in coordinates.

According to embodiments of the present invention, the 3D matching unit triangularizes (xr, yr) and (xl, yl), which are the position values of the 2D artificial marker point obtained from both directions (right and both) in the marker recognition unit. and a three-dimensional transformation unit for reconstructing the 3D marker point values (x, y, z) using a survey method.

According to embodiments of the present invention, the three-dimensional transformation unit arranging the relative positions of the 3D marker points on the basis of the reference marker for external use of the large machine tool having the N multi-points (multi-points) , expressed as each of the 3D marker point values (Xm, Ym, Zm).

According to embodiments of the present invention, when the 3D matching unit connects each of the 3D points reconstructed from the 3D transform unit in the row direction, the overlapping portion of the marker points in the left and right 3D points is horizontally 3D point seam part for stitching, each of the 3D points connected horizontally in the column direction, where the marker points overlap, or for stitching 3D points in the opposite order; further includes

According to embodiments of the present invention, the 3D image seam unit repeats stitching the divided 3D images Y times to generate one whole 3D marker point aggregate.

According to embodiments of the present invention, the 3D coordinate system expression unit expresses the positions of N marker points in each of the M 3D markers derived from the 3D matching unit in a 3D coordinate system.

In a method for generating a new 3D point coordinate system in an apparatus for precise 3D scanning using a multipoint marker and stereo vision according to another embodiment of the present invention,

Taking pictures of a large machine tool attached with a 2D artificial marker from a stereo vision camera; and converting the 2D marker points into 3D marker points; and attaching a reference marker to the outside of the large machine tool to designate the coordinate system reference of the 3D point as a fixed value; and relative rotation and translation of the positions of the 3D marker points based on the external reference marker, and the position values (Xm, Ym, Zm) of each of the marker points. and generating a coordinate system representing

According to the precise 3D scanning method using multi-point markers and stereo vision as described above,

First, it is possible to reliably and quickly detect the change in the position of a dynamic machine in an industrial field with a marker-based point.

Second, large areas of the workpiece can be accurately measured by taking multiple images or increasing the camera resolution.

Third, a new coordinate system can be created using multiple points of a marker.

Fourth, it can be applied to 3D vibration measurement by reconstructing a 3D surface by utilizing the coordinate system unique to the present invention.

1 is a multi-point marker of the present invention.
2 is a flowchart of a method for searching a 3D point using the optical probe of the present invention.
3 is an overall flowchart of the present invention.
4 is a block diagram of a marker recognition unit and a 3D matching unit according to an embodiment of the present invention.
5 is a diagram of a method for generating a 3D coordinate system according to an embodiment of the present invention.
6 is an experimental result diagram and result table according to one embodiment of the present invention.
7 is a 3D stitching method diagram according to an embodiment of the present invention.

A precise 3D scanning apparatus and method using a multi-point marker and stereo vision according to embodiments of the present invention will be described in detail with reference to the accompanying drawings. Since the present invention can have various changes and can have various forms, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements. In the accompanying drawings, the dimensions of the structures are enlarged than the actual for clarity of the present invention, or shown reduced than the actual in order to understand the schematic configuration.

Also, terms such as first and second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. Meanwhile, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

1 is a multi-point marker of the present invention.

Referring to Figure 1, in the case of the present invention, in order to extract information on the large workpiece, a photogrammetry system with artificial markers attached to the large workpiece is described, and several photos of the workpiece are captured by cameras located in various positions and directions. do. Here, the image point of the marker is extracted through image processing technology. From the extracted image points, the spatial relationship between the camera pose and the 3D coordinates of the marker is extracted, and the artificial marker is attached to the large workpiece. The spatial relationship between the two cameras used in the stereo vision system is calculated using Jang's correction method, and the Checkerboard type marker enables the extraction of 9 or more points, which are multi-point from one marker. do. In addition, multi-point markers provide surface local information, curvature and normal vectors, and improve the accuracy of the connection process.

Fig. 2 is a flowchart of a method for searching a 3D point using the optical probe of the present invention, and Fig. 3 is an overall flowchart of the present invention.

2 and 3, a stereo imaging unit 100 for photographing a large machine tool attached with an artificial marker having N multi-points through stereo vision cameras; and a stereo imaging unit 100 A marker recognition unit 200 that searches for the positions of the M markers in the 2D image taken from the large machine tool and represents them in 2D coordinates; and N markers in each of the M markers searched from the marker recognition unit 200 3D matching unit 300 for converting the recognition positions of the points (marker points) into 3D (3-Dimension); and a 3D coordinate system representation unit 400 representing the positions of the MxN 3D marker points derived from the 3D matching unit 300 in a 3D coordinate system. Referring to FIG. 3, the stereo imaging unit 100 captures the artificial markers of the multi-points of 2D (2-Dimension) attached to the surface of the large machine tool, and performs a stereo vision function. at least two stereo vision cameras equipped with; Also, from each of the two or more of the stereo vision cameras, iteratively photographing the large machine tool surface divided by nxm (randomly), wherein the divided area located in the area of each of the divided large machine tool is said Repeat shots from both sides (right and left) of the stereo vision camera. (That is, the two stereo cameras included in the stereo photographing unit 100 take pictures on the right and left sides, respectively, for the divided area.)

4 is a block diagram of the marker recognition unit 200 and the 3D matching unit 300 according to an embodiment of the present invention.

Referring to FIG. 4, the marker recognition unit 200 includes a marker checking unit 205 for recognizing the M number of 2D multi-point markers in the regions of the large machine tool surface obtained from the stereo imaging unit 100; and a marker point search unit 210 for searching the positions of the two-dimensional artificial marker multi-points searched by the marker checking unit 205 . In addition, it further includes; a 2D coordinate representation unit 215 indicating the positions of the 2D marker points searched by the marker point search unit 210 in coordinates. Here, the marker point search unit 210 captures any one of the divided areas of the large machine tool in the stereo photographing unit 100, both (right and left) in the stereo vision camera located in the one area. By searching for the positions of one 2D marker point, the 2D coordinate representation unit 215 expresses the positions of the 2D marker points in both (right and left) directions as (xr, yr) and (xl, yl).

In the case of the 3D matching unit 300, the position values (xr, yr) and (xl, yl) of the 2D artificial marker points obtained from both directions (right and both) in the marker recognition unit 200 are triangulated. A three-dimensional transformation unit 305 for reconstructing a 3D marker point value (x, y, z) using a method; includes, and the three-dimensional transformation unit 305 is the large craft with the N multi-points. Relative positions of the 3D marker points are arranged with respect to an external reference marker of the device, and are expressed as each of the 3D marker point values (Xm, Ym, Zm).

More specifically, after obtaining 2D points from the left and right camera images of the stereo vision, the pair of 2D points are reconstructed into one 3D point through triangulation. In addition, the 3D matching unit 300 connects each of the 3D points reconstructed from the 3D transformation unit 305 in the column direction, horizontally connecting the overlapping portions of the marker points in the left and right 3D points In the case of stitching, each of the 3D points connected horizontally in the row direction is connected to the overlapping part of the marker points, or when each of the 3D points reconstructed in the opposite order are connected in the row direction, In the upper and lower 3D points, the overlapping part of the marker points is vertically stitched, and the part where the marker points overlap in the column direction is horizontally connected to each of the vertical 3D points. Attaching (stitching) 3D point seam 310; further includes. Here, the 3D point seam 310 repeats the stitching of the divided 3D points Y times to generate one whole 3D marker point aggregate, and then 3D in the 3D coordinate system expression unit 400 The positions of the 3D marker point aggregates derived from the matching unit 300 are expressed in a three-dimensional coordinate system.

5 is a diagram illustrating a method of generating a 3D coordinate system according to an embodiment of the present invention.

Referring to Figure 5, in a method of generating a new 3D point coordinate system in a precision 3D scanning device using a multi-point marker and stereo vision, photographing a large machine tool with a 2D artificial marker from a stereo vision camera; and 2D The steps of converting the marker points into 3D marker points; and attaching a reference marker to the outside of the large machine tool so as to designate the coordinate system reference of the 3D point as a fixed value; And based on the external reference marker (reference marker), by calculating the relative rotation (rotation) and movement position (translation) of the positions of the flag 3D marker points, the position values (Xm, Ym, Zm) of each of the marker points and generating a coordinate system representing

Here, detecting 2D marker points in the segmented image is one of the most important processes because it significantly affects the accuracy of 3D position. In all the divided images, the positions of 9 points in the one marker can be found. A second-order differentiation is performed using a linear convolution filter on the marker image to remove irrelevant points. Finally, the position of the marker point (point) of the 2D image is precisely retrieved using a quadratic function with the points around the selected marker point (point). The position of the detected point is rearranged based on the external reference marker point (point). In the present invention, a linear triangulation method is used to find 3D marker points (points), and the 3D position of the point P is two 2D points in different image planes, i.e. (α1 α1)^T and (α2 β2)^ It can be reconstructed using T.

6 is a schematic diagram and result table of experimental results of one embodiment of the present invention.

Referring to FIG. 6 , since the single marker has 9 points, it is a result schematic diagram and result table showing that there is an advantage in forming a three-dimensional coordinate system. Attaching one reference marker having the nine points to the outside of the large machine tool, converting the 2D image photographed with the stereo vision camera into three dimensions, through the three-dimensional stitching process, several large-area workpieces It is possible to process by attaching an image of a dog. Several points in one marker enable coordinate system formation operation through one marker, and are shown in FIG. 6 as points that facilitate generation of position values in the direction perpendicular to the camera when 3D point stitching is performed.

To explain in more detail, case1 is the result of concatenating overlapping marker point parts in 1st horizontally with the marker having 9 points, and then concatenating overlapping marker points vertically in 2nd, that is, 2 lines As a result graph and result table by connecting the overlapping (overlapping) horizontal and vertical parts with Repeatability was derived as 0.044mm as a result of overlapping (overlapping) parts in one line and then overlapping (overlapping) in two lines in the vertical direction. In case 3, using only one point in the middle out of the 9 points of one marker, after concatenating overlapping marker point parts in 1st horizontally, as a result of concatenating overlapping marker points vertically in 2nd, repeatability (Repeatability) was 0.049mm, and the worst performance value was derived. That is, when converting the 3D image, one marker has 9 points, indicating that there is an effect of increasing accuracy in overlapping with two lines. (In the repeatability of z)

7 is a 3D stitching method diagram according to an embodiment of the present invention.

Referring to FIG. 7 , 3D points are connected by overlapping two marker rows and columns with adjacent images. Sharing the above rows and columns line by line reduces the number of required photos, but thereafter, there is a problem in that the error proportionally increases during the sequential stitching process.

7, the 3D point set is sequentially combined in two directions. The first process connects 3D set points aligned in row or column direction. The second process joins the parts stitched in the first process.

To explain in more detail, if 3D points are stitched in the row direction in the first process, the 3D points connected in the first process are connected in the column direction in the second process. Conversely, if the direction of the first process is selected as the column direction, the direction of the second process is the row direction. The direction of the first concatenation process can affect accuracy depending on the number of rows and columns it shares during the stitching process. According to the precise 3D scanning method using multi-point markers and stereo vision as described above, first, it is possible to stably and quickly detect a change in the position of a dynamic machine in an industrial field machine with a marker-based point. Second, large areas of the workpiece can be accurately measured by taking multiple images or increasing the camera resolution. Third, a new coordinate system can be created using the multifocal point of the marker. Fourth, it can be applied to 3D vibration measurement by reconstructing a 3D surface by utilizing the coordinate system unique to the present invention.

In the detailed description of the present invention described above, although it has been described with reference to preferred embodiments of the present invention, those skilled in the art or those having ordinary knowledge in the technical field will be described later in the claims of the present invention And it will be understood that various modifications and variations of the present invention can be made without departing from the technical scope.

100: stereo recording unit 200: marker recognition unit
205: marker checking unit 210: marker point search unit
215: 2D coordinate expression unit 300: 3D matching unit
305: three-dimensional transformation unit 310: 3D image seam
400: 3D coordinate system expression unit

Claims (11)

A stereo imaging unit that takes pictures of a large machine tool attached with artificial markers having N multi-points through stereo vision cameras; and
A marker recognition unit that searches for the positions of the M markers in the 2D image of the large machine tool taken from the stereo imaging unit and displays them in 2D coordinates; and
a 3D matching unit converting the recognition positions of N marker points in each of the M markers searched by the marker recognition unit into 3D (3-Dimension); and
and a 3D coordinate system representation unit representing the positions of the NxM 3D marker points derived from the 3D matching unit in a 3D coordinate system. According to claim 1,
The stereo recording unit,
Photographing the artificial markers of the multi-points of 2D (2-Dimension) attached to the surface of the large machine tool,
A precision 3D scanning device using a multi-point marker and stereo vision, comprising: two or more stereo vision cameras having a stereo vision function. 3. The method of claim 2,
The stereo recording unit,
from each of the at least two stereo vision cameras,
Repeatedly photographing the surface of the large machine tool divided by nxm,
A precision 3D scanning device using a multi-point marker and stereo vision, characterized in that images are taken from both sides (right and left) from the stereo vision camera located in each of the divided areas. According to claim 1,
The marker recognition unit,
a marker checking unit for recognizing the M number of 2D multi-point markers in the areas of the surface of the large machine tool obtained from the stereo imaging unit; and
A precision 3D scanning device using multi-point markers and stereo vision, comprising: a marker point search unit for searching the positions of the multi-points of the two-dimensional artificial markers found from the marker checking unit. 5. The method of claim 4,
The marker recognition unit,
Precision 3D scanning device using multi-point markers and stereo vision, characterized in that it further comprises; According to claim 1,
The 3D matching unit,
The 3D marker point value (x, A precision 3D scanning device using a multi-point marker and stereo vision, characterized in that it comprises; 7. The method of claim 6,
The three-dimensional transformation unit,
Based on a reference marker attached to the outside of the large machine tool having the N multi-points,
By arranging the relative positions of the 3D marker points,
A precision 3D scanning device using a multi-point marker and stereo vision, characterized in that it is expressed as each of the 3D marker point values (x, y, z). 7. The method of claim 6,
The 3D matching unit,
When each of the 3D points reconstructed from the three-dimensional transformation unit is connected in the row direction,
Horizontally stitching the part where the left and right marker points overlap,
Each of the 3D points connected horizontally is connected to the overlapping part of the marker points in the column direction,
Or a 3D point seam for stitching 3D points in the opposite order to this; precision 3D scanning device using a multi-point marker and stereo vision, characterized in that it further comprises. 9. The method of claim 8,
The 3D point seam part,
Repeating the stitching of the divided 3D points Y times,
A precision 3D scanning device using a multi-point marker and stereo vision, characterized in that it creates an aggregate of all 3D marker points. According to claim 1,
The 3D coordinate system expression unit,
A precision 3D scanning apparatus using a multi-point marker and stereo vision, characterized in that the positions of N marker points in each of the M 3D markers derived from the 3D matching unit are expressed in a three-dimensional coordinate system. A method of generating a new 3D point coordinate system in a precision 3D scanning device using a multipoint marker and stereo vision, the method comprising:
Taking a large machine tool attached with a 2D artificial marker from a stereo vision camera; and
converting 2D marker points into 3D marker points; and
attaching a reference marker to the outside of the large machine tool so as to designate the coordinate system reference of the 3D point as a fixed value; and
Based on the external reference marker (reference marker),
generating a coordinate system representing the position values (Xm, Ym, Zm) of each of the marker points by calculating the relative rotation and translation of the positions of the 3D marker points; A precision 3D scanning method using multi-point markers and stereo vision.

Images