TWI814680B - Correction method for 3d image processing - Google Patents

Abstract

A correction method for three-dimensional image processing, wherein a host obtains a first image and a second image of the image area through a first image acquisition device and a second image acquisition device, the first image and the second image containing a first image object and a second image object of the correction block, respectively. The host obtains a first parameter matrix and a second parameter matrix based on the first image and the second image, respectively, and The first image, the second image, the first image object, the second image object, the first parameter matrix, and the second parameter matrix are used to reconstruct and correct a three-dimensional space, wherein the first parameter matrix and the second parameter matrix correspond to the correction functions of the first imaging device and the second imaging device, respectively, to provide depth information for the three-dimensional space correction.

Description

Correction method for 3D image processing

The present invention relates to a correction method for three-dimensional image processing, particularly a correction method for reconstructing a three-dimensional image so that the correction includes depth information.

Modern technology develops rapidly. With the increasingly sophisticated manufacturing processes and the popularization of high-speed computers, the demand for high-precision measurement has greatly increased. In the pursuit of automated factory production lines, there are often various sensors that transmit signals back to computers. , with the upgrade of photosensitive components and computer hardware, computer vision (Computer Vision) has gradually become one of the important technologies for factory automation.

Computer vision refers to the use of image processing technology to extract information from images for measurement, tracking, identification, etc. It is a type of automated optical inspection (Automated Optical Inspection, AOI). This technology has been widely used in various engineering fields. Compared with Compared with traditional measurement methods, this technology has the advantages of high precision, non-contact measurement, and full-area measurement.

Digital Image Correlation (DIC) is a full-field non-contact optical measurement method that calculates the correlation between images to obtain the displacement of the object under test, and uses the displacement information to match the object under test The material coefficient of the object can be used to obtain stress and strain related information. Digital image information is stored in matrix form. The smallest unit is pixel (Pixel). In grayscale images, the brightness of each pixel is composed of 8 bits (Bit), and the brightness ranges from 0 to 255, and Will be based on pixels The positions correspond to the values in the matrix, forming a two-dimensional array. Color images are stored in a three-dimensional array, with the third dimension representing the brightness values of red (R), G (green), and B (blue) respectively.

The Stereo digital image correlation (Stereo DIC) method uses two parallel cameras to capture the surface of an object. Through 2D DIC combined with the triangulation method, the out-of-plane deformation problem of the cantilever beam can be successfully measured. Through the ideal pin-hole imaging model of two cameras, known points in three-dimensional space are used with the nonlinear least squares method to calculate the internal and external parameter matrices of the camera, and actually apply it to the displacement of the object surface Measurement.

The most widely used correction method for camera calibration is to take several images and use the known geometric length of the correction plate to match the image length of the detection position on the image coordinates to convert the relationship between space and image. Commonly used correction methods The boards are divided into dot correction boards and checkerboard corner correction boards. The checkerboard correction board uses corner point detection to find the image coordinates, while the dot correction board obtains the image coordinates by fitting the center coordinates of the circle. However, because the correction board It is plane. When converting the relationship between spatial coordinates and image coordinates, since there is only x and y axis information, the z-axis coordinate is assumed to be zero. However, when the correction plate is actually photographed, the z-axis coordinate is not zero due to the rotation of the correction plate. , so this correction method is prone to errors.

The coordinate system used in traditional space reconstruction is used to define the coordinate system on the left camera. The information that can be learned from the correction process is the position of the left and right cameras, and the position of the correction tool. Since the correction plate does not have depth information, 3D reconstruction The left camera is usually used as the main coordinate system, and the coordinate plane of the camera is parallel to the imaging plane. Therefore, the left camera must be parallel to the object to be measured during measurement, otherwise it will become unintuitive due to the different definitions of the coordinate system. , the angle between the camera and the object to be measured must be obtained first and then the actual data can be known through coordinate conversion. When more cameras are used, coordinate conversion will become quite complicated and troublesome.

To this end, it is necessary to provide a correction method for three-dimensional image processing that can provide depth information for correction without defining a coordinate system on the camera. This is a problem that those skilled in the art want to solve.

One purpose of the present invention is to provide a correction method for three-dimensional image processing, which performs correction containing depth information through correction blocks to reduce errors generated during correction. Another purpose of the present invention is to provide a correction method for three-dimensional image processing. , when reconstructing the three-dimensional space, it sets the image of the correction block as the origin of the world coordinate system, so that when multiple imaging devices capture images at the same time, there is no need to define the coordinate system on the camera, and it does not require a large number of When converting the coordinate system, there is no need to control the imaging device to remain parallel to the object to be measured.

To achieve the above objectives, the present invention provides a correction method for three-dimensional image processing, in which a first imaging device and a second imaging device are provided. The first imaging device locks an image area at a first imaging angle. And connected to a host, the second imaging device locks the image area at a second imaging angle, is connected to the host, and configures a correction block in the image area. The steps include: the host uses the The first imaging device and the second imaging device obtain a first image and a second image of the image area respectively, and the first image and the second image respectively include a first image object of the correction block. and a second image object; the host obtains a first parameter matrix and a second parameter matrix respectively based on the first image and the second image; the host reconstructs a three-dimensional space based on the first image and the second image, The three-dimensional space includes a spatial object corresponding to the correction block; and the host corrects the three-dimensional space according to the first image object, the second image object, the first parameter matrix and the second parameter matrix; wherein , the correction block is a rectangular parallelepiped, and each side of the correction block Both are provided with a checkerboard pattern, and the checkerboard pattern corresponds to a depth information. Furthermore, the first parameter matrix and the second parameter matrix respectively correspond to the device parameters of the first imaging device and the second imaging device, This makes the correction when reconstructing the three-dimensional space simpler and faster, and also reduces the error during correction.

The present invention provides an embodiment, in which the step of obtaining a first parameter matrix and a second parameter matrix respectively based on the first image and the second image further includes: the host uses a corner point detection method to obtain the third parameter matrix respectively. An image object is located at a plurality of corner points in the first image, and the second image object is located at a plurality of corner points in the second image; the host obtains corresponding spatial coordinate information based on the corner points. ; The host obtains the first parameter matrix based on the spatial coordinate information and the first image, and the host obtains the second parameter matrix based on the spatial coordinate information and the second image.

The present invention provides an embodiment, in which the corner point detection method determines the grayscale value of the first image or the second image through the host, and when the grayscale values in two main directions change significantly, it is the corner point. , when the grayscale value in only one main direction changes significantly, it is an image edge, and when there is no change in the grayscale value, it is a uniform area.

The present invention provides an embodiment, in which the host obtains the first parameter matrix based on the spatial coordinate information and the first image, and the host obtains the second parameter matrix based on the spatial coordinate information and the second image, The host further uses singular value decomposition to calculate the spatial coordinate information and the first image and the second image to obtain a first parameter matrix and a second parameter matrix.

The present invention provides an embodiment in which the host reconstructs a three-dimensional space based on the first image and the second image, further including: the first image and the second image reconstruct the three-dimensional space using a digital image correlation method. .

The present invention provides an embodiment in which the first imaging device and the second imaging device are digital cameras or smart phones.

The present invention provides an embodiment, in which the step of the host correcting the three-dimensional space through the first parameter matrix and the second parameter matrix further includes: when the host corrects the three-dimensional space, the host converts an original value of the three-dimensional space The point is set on this space object.

The present invention provides an embodiment in which the first parameter matrix and the second parameter matrix are homography matrices.

S1~S7: steps

S31~S35: steps

10:Host

20: First imaging device

22:First image

222: First image object

2222:Side 4

2224:The fifth side

2226:Sixth side

30: Second imaging device

32: Second image

322: Second image object

3222:Seventh side

3224:Side 8

3226:Ninth side

40:Image area

50: Correction block

502: Side 1

504:Second side

506:The third side

60: Three-dimensional space

602: Space objects

70:Corner point

72: Space coordinate information

722: First parameter matrix

724: Second parameter matrix

The first figure: It is a schematic flow diagram of an embodiment of the present invention; The second figure: It is a schematic flow diagram of an embodiment of the present invention; The third figure: It is a schematic diagram of the system architecture of an embodiment of the present invention; Figure 4A: It is a schematic diagram of the correction block model of one embodiment of the present invention; Figure 4B: It is a schematic diagram of the conventional correction plate model; Figure 5: It is the first image and the third image of one embodiment of the present invention. Two image diagrams; the sixth diagram: which is a three-dimensional spatial diagram of an embodiment of the present invention; and the seventh diagram: which is a schematic diagram of the first parameter matrix and the second parameter matrix of one embodiment of the present invention.

In order to enable your review committee to have a further understanding of the characteristics and effects of the present invention, we would like to provide preferred embodiments and accompanying detailed descriptions, which are as follows: In view of the fact that the correction convenience of conventional three-dimensional image processing is insufficient, which results in the need for a large number of coordinate system conversions and the need to place the imaging device parallel to the object to be measured. Accordingly, the present invention proposes a correction method for three-dimensional image processing. To solve the problem of insufficient correction convenience in conventional three-dimensional image processing.

This invention improves the previous use of correction plates to perform error correction on the imaging device required for three-dimensional space reconstruction. Through the self-made correction block, the correction process can produce more depth information than before, thereby reducing the errors caused by the imaging device and making the The reconstruction of three-dimensional images can be closer to reality.

In the following, the present invention will be described in detail by illustrating various embodiments of the present invention through drawings. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the illustrative embodiments set forth herein.

First, please refer to the first figure, which is a flow chart of an embodiment of the present invention. As shown in the figure, the steps of the correction method for three-dimensional image processing of the present invention include:

Step S1: A host obtains a first image and a second image of an image area through a first imaging device and a second imaging device respectively, and the first image and the second image respectively include the correction block a first image object and a second image object.

Step S3: The host obtains a first parameter matrix and a second parameter matrix based on the first image and the second image respectively.

Step S5: The host reconstructs a three-dimensional space based on the first image and the second image. The three-dimensional space includes a spatial object, and the spatial object corresponds to the correction block.

Step S7: The host corrects the three-dimensional space according to the first image object, the second image object, the first parameter matrix and the second parameter matrix. .

The system architecture of one embodiment of the present invention, as shown in the third figure, includes: a host 10; a first imaging device 20, the first imaging device 20 is locked at a first imaging angle An image area 40 and connected to the host 10; a second imaging device 30, the second imaging device 30 locks the image area 40 with a second imaging angle and connected to the host 10; configure a correction In block 50 , in the image area 40 , a first image 22 is transmitted from the first imaging device 20 to the host 10 , and a second image 32 is transmitted from the second imaging device 30 to the host 10 .

Next, each step will be detailed and explained as follows; In step S1, as shown in the first figure, in this embodiment, a host 10 obtains a first image 22 and a first image 22 of an image area 40 through a first imaging device 20 and a second imaging device 30 respectively. The second image 32, the first image 22 and the second image 32 respectively include a first image object 222 and a second image object 322 of the correction block 50, wherein, as shown in the fourth figure A, the correction Each side of the block 50 is provided with a checkerboard pattern, and the checkerboard pattern corresponds to a depth information, which is the depth of field of the image.

In step S3, as shown in the first figure, in this embodiment, the host 10 obtains a first parameter matrix 722 and a second parameter matrix 724 respectively based on the first image and the second image, wherein the first parameter The matrix 722 and the second parameter matrix 724 respectively correspond to the device parameters of the first imaging device 20 and the second imaging device 30 .

In step S5 , in this embodiment, the host 10 reconstructs a three-dimensional space 60 based on the first image 22 and the second image 32 . The three-dimensional space 60 includes a spatial object 602 , and the spatial object 602 corresponds to the correction block 50 , wherein the host 10 reconstructs the three-dimensional space 60 through a digital image correlation method.

In step S7, in this embodiment, the host 10 corrects the three-dimensional space 60 based on the first image object 222, the second image object 322, the first parameter matrix 722 and the second parameter matrix 724, wherein when the host 10. When calibrating the three-dimensional space 60, the host 10 sets an origin of the three-dimensional space 60 on the spatial object 602.

Step S3 further includes step S31, step S33 and step S35, which are described as follows: Next, refer to the second figure, which is a flow chart of another embodiment of the present invention. As shown in the figure, this embodiment is based on the above embodiment, and the steps of the correction method for three-dimensional image processing of the present invention are:

Step S31: The host uses a corner point detection method to respectively obtain the plurality of corner points 70 of the first image object 222 located in the first image 22, and the plurality of corner points 70 of the second image object 322 located in the second image 32. Plural corner points 70.

Step S33: The host obtains corresponding spatial coordinate information 72 based on the corner points 70 .

Step S35: The host obtains the first parameter matrix 722 based on the spatial coordinate information 72 and the first image 22, and the host obtains the second parameter matrix 724 based on the spatial coordinate information 72 and the second image 32.

In step S31, this embodiment uses a corner point detection method on the host 10 to respectively obtain a plurality of corner points 70 of the first image object 222 located in the first image 22, and a plurality of corner points 70 of the second image object 322 located in the first image 22. There are a plurality of corner points 70 in the second image 32. The corner point detection method is to determine the grayscale value of the first image 22 or the second image 32 through the host 10. When there are two main directions of grayscale, When the level value changes significantly, it is the corner point 70. When the gray level value in only one main direction changes significantly, it is an image edge. When there is no change, it is a uniform area.

In step S33 , in this embodiment, the host 10 obtains corresponding spatial coordinate information 72 based on the corner points 70 .

In step S35, in this embodiment, the host 10 obtains the first parameter matrix 722 based on the spatial coordinate information 72 and the first image 22, and the host 10 obtains the third parameter matrix 722 based on the spatial coordinate information 72 and the second image 32. Two parameter matrices 724, wherein the first parameter matrix 722 and the second parameter matrix 724 respectively correspond to the device parameters of the first imaging device 20 and the second imaging device 30.

Next, the following actual example is given to illustrate the actual use example of the correction of the three-dimensional image processing in this embodiment, but it is not limited to this.

In the following embodiments, a digital camera is used as an example of an imaging device, but the actual application is not limited to this.

In the following embodiments, the first imaging angle is 0 degrees relative to the correction block, which is used as an example of the imaging device, and the actual application is not limited thereto.

First, a left camera (the first imaging device 20) and a right camera (the second imaging device 30) respectively take pictures of a correction block 50 in an area (the image area 40), and respectively capture the image (the first image 22. The second image 32) is uploaded to a host 10, and the host 10 uses a corner detection method to detect the correction block 50 in the image.

Among them, as shown in Figure 4A and Figure 5, the first image object 222 and the second image object 322 are through the first imaging angle (0 degree angle relative to the correction block) and the second image object 322 respectively. The image of the correction block 50 is captured at an imaging angle (45 degrees relative to the correction block), and the first surface 502 of the correction block 50 corresponds to the fourth surface 2222 of the first image object 222 and the second image object 322 The seventh side 3222 of the correction block 50 corresponds to the fifth side 2224 of the first image object 222 And the eighth surface 3224 of the second image object 322, the third surface 506 of the correction block 50 corresponds to the sixth surface 2226 of the first image object 222 and the ninth surface 3226 of the second image object 322.

Since the corner point detection target in this embodiment is the checkerboard correction block 50, the corner point detection method in this embodiment utilizes the fact that when the grayscale values of feature points in two directions suddenly change significantly, it is very likely that they are corner points. 70. If the grayscale values in both directions of the pane moving in all directions change, it is considered a corner point 70; if there is no change in both directions, it is a uniform area; if there is a change in only one direction, then Identified as the edge of the image. Suppose I ( x, y ) is the grayscale value of the image to be detected, ( x, y ) is the point in the local pane, E ( u, v ) is the sum of squared pixel differences before and after the pane is translated, as Expressed by formula (1).

E ( u, v ) = Σ _x Σ _y w ( x, y ) [ I ( x + u, y + u ) - I ( x, y )] ² Formula (1)

where w ( x, y ) represents the weighting function of the pane. The size of the pane can be changed according to the size of the image. The larger the coefficient of the weighting function, the greater the weight of the grayscale value difference on the result. Usually, the mean function or Gaussian function filters where the corners are. Then do the first-order Taylor expansion to get equation (2) I ( x + u,y + v ) = I ( x,y ) + I _x ( x,y ) u + I _y ( x,y ) vEquation (2) )

After substituting formula (2) into (1), we can get formula (3)

Where M ( x,y ) is formula (4)

Considering that the boundary of the corner point is usually consistent with the coordinate axis, as shown in Figure 3-10, in the pane, only the left side follows the upper side. I _x is large and I _y is small on the left side, and I y is large and I _x is _small on the upper side. It can be M is simplified to equation (5)

For the M matrix, it can be compared with the covariance matrix. The covariance matrix represents the correlation between multi-dimensional random variables. Its diagonal elements represent the variance of each dimension. Diagonalizing it can reduce the difference. Dimensional correlation and obtain the places with larger eigenvalues. The matrix M can be regarded as a two-dimensional randomly distributed covariance matrix, and the corner points can be determined by diagonalizing the two eigenvalues obtained.

Finally, when judging whether it is a corner point, you only need to calculate the response value R of the approximate corner point as shown in equation (6): R =| M |- α ( traceM ) ² = λ ₁ λ ₂ - α ( λ ₁ + λ ₂ ) ² formula(6)

[0.04,0.06]，當響應值大於設定閥值時即判定為角點。 α in formula (6)

[0.04,0.06], when the response value is greater than the set threshold, it is determined to be a corner point.

Then, the corner point is detected through the corner point detection method and the spatial coordinate information corresponding to the corner point is obtained, and then the spatial coordinate information is converted into the first parameter matrix of the left camera and the second parameter matrix of the right camera.

In this embodiment, a correction block is used instead of a correction plate. Through the pinhole imaging principle and the relationship between camera coordinates and world coordinates, the first parameter matrix and the second parameter matrix of the left camera and the right camera can be obtained using equations (7) to (11). parameter matrix.

平移矩陣，

為影像座標以及

為世界座標，並將式(7)改寫為式(8)。 Among them, K is the camera device parameter, R is the rotation matrix,

translation matrix,

are the image coordinates and

are the world coordinates, and rewrite equation (7) into equation (8).

，若各有N個座標點則可將式(8)改寫為式(9)。 H is called the Homography Matrix, with a size of 3×4. The image coordinate point position ( u _i , v _i , 1) ^T obtained by actual measurement is matched with the known world coordinate position.

, if each has N coordinate points, Equation (8) can be rewritten as Equation (9).

Then Equation (9) can be expanded and rewritten into Equation (10).

Write equation (10) into matrix form and organize it into equation (11).

分別為式(12)和式(13)

where L and

They are formula (12) and formula (13) respectively.

Then, by using singular value decomposition (SVD) to solve equation (11), the first parameter matrix and the second parameter matrix of the left camera and the right camera can be obtained.

Finally, as shown in Figure 6, the host reconstructs a three-dimensional space 60 based on the stereoscopic digital image correlation method based on the images captured by the two cameras, and corrects the three-dimensional space 60 through the obtained first parameter matrix 722, and the The host 10 sets an origin of the three-dimensional space 60 on the spatial object 602 .

In the embodiments described above, the method of the present invention is different from the past. The conventional correction method uses a correction plate for correction. However, the correction plate only has plane space information. As shown in Figure 4B, when calculating the parameters of the camera When matrixing, it is often assumed that the depth coordinate of the photographed correction plate is zero, and then the camera is calibrated through several correction plates with different postures. The present invention proposes to use the correction block method, because the correction block has three-dimensional spatial information, and there is no need to assume the depth coordinate. is zero, and because the three faces on the correction block each have a checkerboard pattern, the present invention only needs one photo to correct the parameter matrix of the camera, and because the depth coordinate is not zero, it can be re-projected through the reconstructed three-dimensional space after correction The error in the two-dimensional image coordinates can be much smaller than the method of using a correction plate, and the present invention establishes the coordinate system on the correction block. As long as cameras from different angles can capture the checkerboard pattern on the correction block, the camera's parameter matrix can be corrected. Conventional camera calibration requires one-to-one conversion to the main coordinate system through the rotation and translation relationships between cameras, which will lead to The steps when aligning the camera are very complicated and errors are prone to occur during coordinate system conversion. This method of directly establishing the coordinate system on the object to be measured can avoid the need to use the coordinate system of one camera as the main coordinate system when calibrating multiple cameras. problem.

Therefore, this invention is indeed novel, progressive and can be used industrially. It should undoubtedly comply with the patent application requirements of my country’s Patent Law. I file an invention patent application in accordance with the law and pray that the Office will grant the patent as soon as possible. I am deeply grateful.

However, the above are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention. All changes and modifications can be made equally in accordance with the shape, structure, characteristics and spirit described in the patent scope of the present invention. , should be included in the patent scope of the present invention.

S1~S7: steps

Claims (8)

A correction method for three-dimensional image processing, in which a first imaging device and a second imaging device are provided. The first imaging device locks an image area at a first imaging angle and is connected to a host. The first imaging device locks an image area at a first imaging angle and is connected to a host. Two imaging devices lock the image area at a second imaging angle, connect to the host, and configure a correction block in the image area. The steps include: The host obtains a first image and a second image of the image area through the first imaging device and the second imaging device respectively, and the first image and the second image respectively include one of the correction blocks. a first image object and a second image object; The host obtains a first parameter matrix and a second parameter matrix respectively based on the first image and the second image; The host reconstructs a three-dimensional space based on the first parameter matrix and the second parameter matrix, the three-dimensional space includes a spatial object, and the spatial object corresponds to the correction block; and The host corrects the three-dimensional space according to the first image object, the second image object, the first parameter matrix and the second parameter matrix; Wherein, the correction block is a rectangular parallelepiped, and each side of the correction block is provided with a checkerboard pattern. The checkerboard pattern corresponds to a depth information. Furthermore, the first parameter matrix and the second parameter matrix respectively correspond to the depth information. Device parameters of the first imaging device and the second imaging device. The correction method for three-dimensional image processing as described in claim 1, wherein the step of obtaining a first parameter matrix and a second parameter matrix respectively based on the first image and the second image further includes: The host uses a corner detection method to respectively obtain the plurality of corner points of the first image object located in the first image, and the plurality of corner points of the second image object located in the second image; The host obtains corresponding spatial coordinate information based on the corner points; The host obtains the first parameter matrix based on the spatial coordinate information and the first image, and obtains the second parameter matrix based on the spatial coordinate information and the second image. The correction method for three-dimensional image processing as described in claim 3, wherein the corner detection method is to determine the grayscale value of the first image or the second image through the host. When there are grayscale values in two main directions When there is a significant change, it is a corner point. When there is a significant change in the grayscale value in only one main direction, it is an image edge. When there is no change, it is a uniform area. The correction method for three-dimensional image processing as described in claim 3, wherein the host obtains the first parameter matrix based on the spatial coordinate information and the first image, and the host obtains the third parameter matrix based on the spatial coordinate information and the second image. In the two-parameter matrix step, the host further uses singular value decomposition to calculate the spatial coordinate information and the first image and the second image to obtain the first parameter matrix and the second parameter matrix. As for the three-dimensional image processing correction method described in claim 1, the step of reconstructing a three-dimensional space by the host based on the first image and the second image further includes: The first image and the second image use a digital image correlation method to reconstruct the three-dimensional space. As for the correction method of three-dimensional image processing described in claim 1, the first imaging device and the second imaging device are digital cameras or smart phones. As for the correction method of three-dimensional image processing described in claim 1, the step of correcting the three-dimensional space by the host through the first parameter matrix and the second parameter matrix further includes: When the host corrects the three-dimensional space, the host sets an origin of the three-dimensional space on the spatial object. As claimed in claim 1, the correction method for three-dimensional image processing, wherein the first parameter matrix and the second parameter matrix are homography matrices.

Images