WO2022052313A1 - Calibration method for 3d structured light system, and electronic device and storage medium - Google Patents

Abstract

A calibration method for a 3D structured light system, the method comprising: manufacturing a calibration board; collecting calibration board images; and calibrating a structured light system according to the collected plurality of calibration board images. Further provided are an electronic device and a storage medium.

Description

Calibration method, electronic device and storage medium for 3D structured light system

The present disclosure claims the priority of a Chinese patent application with application number 202010951187.9 filed with the Chinese Patent Office on September 11, 2020, the entire contents of which are incorporated into the present disclosure by reference.

technical field

The present application relates to the field of computer vision technology, for example, to a calibration method of a 3D structured light system, an electronic device, and a storage medium.

Background technique

Among related technologies, 3D structured light imaging technology is mainly used in industries such as precision measurement and defect detection, covering manufacturing fields such as surface assembly technology, automotive, aviation, semiconductor, medical, pharmaceutical, and food processing. Compared with two-dimensional detection, 3D detection technology provides one-dimensional height information. In the actual measurement process, it can not only accurately locate the measured object, accurately measure two-dimensional information such as size and color, but also obtain the surface contour information of the object. , which can measure and analyze complex information such as flatness, slope, curvature and flaws of the surface of the object. 3D structured light measurement has a series of advantages such as non-contact, fast speed, high precision and strong anti-interference ability, and is a very important direction of precision measurement in the 3D inspection industry.

The 3D structure light calibration algorithm determines the detection accuracy of the 3D measurement system. The 3D structure light calibration algorithm is mainly an algorithm to establish the internal parameters of the camera, the structured light projection and the external parameters of the system. It is only by determining these system model parameters. The 3D depth information of the measured object is recovered.

In the related art, the calibration principle of structured light 3D system adopts Zhang Zhengyou's calibration method, but in the specific implementation, the methods are not the same, of course, the accuracy difference is also very large, providing a higher-precision calibration method has gradually become a research field. .

In Chinese patent application document CN110443856A, a method for calibrating a 3D structured light module is disclosed. The method includes the following steps: S0, making a calibration board, and printing a checkerboard on the calibration board, and the checkerboard covers the entire calibration board; S1 , Take a calibration photo, place the 3D structured light module in front of the plane of the calibration plate, fix the relative position of the 3D structured light module and the calibration plate, and sequentially take the infrared photo and dot matrix of the calibration plate Projected to the dot matrix projection photo of the calibration plate, the infrared photo of the calibration plate and the dot matrix projection photo are taken as a group of calibration photos; S2, take a calibration photo group, replace the 3D structured light module and all Describe the relative position of the calibration plate, repeat step S1 to take multiple sets of calibration photos; S3, calibrate the 3D structured light module, use a calibration algorithm to calibrate the infrared camera through multiple sets of the calibration photos, and then calibrate the dot matrix projector, Obtain the geometric relationship of the dot matrix projector relative to the infrared camera. The calibration plate is a calibration plate with a white background, the shape of the checkerboard is square, and the gray scale of the black grids in the checkerboard is adjustable; the calibration algorithm is the Zhang Zhengyou plane calibration method; the calibration process includes: S31, extracting Pixel coordinates, extracting the pixel coordinates of the checkerboard grid points of the infrared photo of the calibration board, and setting the corresponding world coordinate values; S32, calibrating the infrared camera, calculating the homography mapping from the world coordinates of the checkerboard grid points to the pixel coordinates, according to the rotation The constraint relationship of the matrix is calculated to obtain the corresponding internal parameters and external parameters of the infrared camera; S33, calculate the conversion relationship, calculate the conversion relationship between the spot coordinates projected by the dot matrix and the dot matrix geometric relationship of the dot matrix projector, and convert the infrared photo. The pixel coordinates of the checkerboard points are converted into the pixel coordinates of the dot matrix projector, and the internal and external parameters of the dot matrix projector are calculated by using the Zhang Zhengyou plane calibration algorithm. This scheme uses a checkerboard calibration board, and needs to detect the corner points of the calibration board as Mark points. In actual calibration applications, when the calibration board poses changes, it will be affected by illumination and uneven imaging quality. The corner extraction accuracy is easily affected, and the circle calibration plate can be used to extract the circle well, and it is not easy to be disturbed.

In Chinese patent application document CN110827357A, a combination pattern calibration plate and a structured light camera parameter calibration method are disclosed. The pattern on the calibration plate includes an identification mark and a code block area; the identification mark is a closed quadrilateral distinguished by black and white boundaries, and its Inside the boundary are black pixels, and outside the boundary are white pixels; the code block area is located inside the quadrilateral boundary, and the code block is encoded information composed of N×N black and white square color blocks. The calibration method includes the following steps: S1: using a plurality of combination pattern calibration boards in claim 1, each combination pattern calibration board has a code block area with different patterns; S2: collecting images of the multiple combination pattern calibration boards distributed and arranged, and Perform preprocessing; S3: Calculate the local gradient of each pixel of the image; set the pixel area with the same gradient magnitude and direction as the connected domain; S4: Use the connected domain with the gradient greater than the set threshold in the connected domain as the edge of the calibration pattern, The edge pixels are fitted into different line segments using straight lines; S5: traverse all line segments, detect whether four adjacent line segments can form a complete quadrilateral, and detect whether all areas with complete quadrilaterals contain valid coding areas; S6: Match the detected code with the pattern of the code library, calculate the coordinates of the pattern corresponding to the code in the calibration plate, and the coordinates of the intersection of the four line segments of the quadrilateral in the image; S7: According to the pixel coordinates in the image and the corresponding calibration The board coordinates are used to calibrate the parameters of the structured light camera. In this scheme, the orientation of the calibration plate is not easy to be tilted too much, and if it is too large, the pattern cannot be matched.

The related technologies have at least the following deficiencies:

1. The accuracy of establishing the correspondence between the camera sensor and the projector sensor is insufficient.

2. The state of the obtained calibration image is limited and cannot fully describe the system model.

SUMMARY OF THE INVENTION

The present application provides a calibration method, electronic device and storage medium for a 3D structured light system. The method uses a calibration board including a dot pattern with higher precision and a more complex calibration process, and uses a multi-frequency phase-shift code with higher precision. To find the corresponding relationship between the camera and the projection sensor, improve the phase accuracy locally, and more accurately determine the positional relationship between the sensors. At the same time, the flexible automatic keystone correction algorithm is used to sort the dots, and the calibration board can basically be placed in various postures. Extract more calibration data.

The present application provides a method for calibrating a 3D structured light system, comprising the following steps:

make a calibration board;

acquiring an image of the calibration plate; and

calibrate the structured light system according to the collected images of the calibration plate;

Wherein, the step of making a calibration board includes: making a system calibration board and burning it, the pattern on the calibration board is a plurality of circular targets, the center distances of the adjacent circular targets are the same, and a plurality of the circular targets The targets are arranged in rows and columns, and the circular targets are of different sizes;

Wherein, the step of collecting the calibration plate image includes:

Set a calibration plate attitude, project sinusoidal grating fringe patterns of multiple frequencies in the horizontal and vertical directions on the calibration plate respectively, project light without grating fringes on the other calibration plate, and place it on the calibration plate. At the same time of each projection, the camera is used to collect the images of the calibration plate projected with various patterns in the posture; and

Adjust the posture of the calibration plate, repeat the steps of collecting the images of the calibration plate, until the number of postures of the calibration plate reaches a preset target value, and obtain a plurality of images of the calibration plate;

Wherein, the step of calibrating the structured light system according to a plurality of the collected calibration plate images includes:

extracting the center positions of all the circular targets in the calibration plate image;

Sort all circle centers according to the extracted center positions of all the circular targets;

The absolute phase value of each circle center coordinate is obtained by bilinear interpolation algorithm;

Using the obtained absolute phase value of the coordinates of each circle center, the coordinates in the projector image corresponding to the coordinates of each circle center in the camera image are obtained;

According to the obtained coordinates of the circle center in the camera and the projector image, the internal parameters, distortion coefficients and external parameters of the camera and the projector are obtained by Zhang Zhengyou's calibration method;

Using the external parameters of the camera and the projector under the same calibration board attitude, calculate the transformation matrix between the camera and the projector coordinate system under each calibration board attitude; and

According to the obtained transformation matrix under the attitude of each calibration board, the calibrated camera and projector coordinate system transformation matrix is obtained.

The application also provides an electronic device, comprising:

processor;

memory, set to store programs,

When the program is executed by the processor, the processor implements the method for calibrating a 3D structured light system as described above.

The present application also provides a computer-readable storage medium storing computer-executable instructions, where the computer-executable instructions are used to execute the calibration method for a 3D structured light system as described above.

Description of drawings

1 is a flow chart of a calibration method of an optional embodiment of the present application;

2 is a flow chart of calibration steps of an optional embodiment of the present application;

3 is a schematic diagram of the sorting of dot patterns in an angled calibration image according to an optional embodiment of the present application;

FIG. 4 is a schematic structural diagram of an electronic device according to an optional embodiment of the present application.

detailed description

The specific embodiments of the present application will be described below with reference to the accompanying drawings 1-3.

The present application provides a method for calibrating a 3D structured light system, comprising the following steps:

The step of making a calibration board is to make a system calibration board and burn it. The pattern on the calibration board is a plurality of circular targets, the center distances of the adjacent circular targets are the same, and the plurality of circular targets are arranged in rows and columns, the circular targets are of different sizes;

Calibration plate image acquisition steps,

Set a calibration plate attitude, project sinusoidal grating fringe patterns of multiple frequencies in the horizontal and vertical directions on the calibration plate respectively, project light without grating fringes on the other calibration plate, and put it on the calibration plate. At the same time of each projection, the camera is used to collect the calibration plate images of various patterns projected under the posture;

Adjust the attitude of the calibration board, repeat the steps of image acquisition of the calibration board, until the number of attitudes of the calibration board reaches the preset target value, and obtain a plurality of images of the calibration board;

The calibration step is to calibrate the structured light system according to a plurality of the collected calibration plate images, which may include the following steps:

Extract the center positions of all circular targets in the calibration plate image;

Sort all the circle centers according to the center positions of all the extracted circular targets;

The absolute phase value of each circle center coordinate is obtained by bilinear interpolation algorithm;

Using the obtained absolute phase value of the coordinates of each circle center, the coordinates in the projector image corresponding to the coordinates of each circle center in the camera image are obtained;

According to the center coordinates in the obtained camera and projector images, the internal parameters, distortion coefficients and external parameters of the camera and projector are obtained by Zhang Zhengyou's calibration method;

Using the external parameters of the camera and the projector under the same calibration board attitude, calculate the transformation matrix between the camera and the projector coordinate system under each calibration board attitude;

According to the obtained transformation matrix under the attitude of each calibration board, the calibrated camera and projector coordinate system transformation matrix is obtained. The parameters of the corresponding positions in all the obtained transformation matrices are superimposed, summed and averaged, as the final transformation matrix of the projector and camera coordinate systems.

The parameters obtained by calibration include: camera internal parameters and distortion coefficients, projector internal parameters and distortion coefficients, and rigid transformation relationship matrix between camera and projector coordinate systems.

As an optional implementation manner, the method for calibrating the 3D structured light system further includes: before using a bilinear interpolation algorithm to obtain the absolute phase value of each circle center coordinate, using a multi-frequency multi-step phase shift algorithm to obtain the plane phase of the calibration plate The absolute phase value of the field in the horizontal and vertical directions. In one embodiment, ten-step phase-shift code and three-frequency heterodyne method can be used to obtain absolute phase values in the horizontal and vertical directions, including the following steps:

In the calibration image acquisition step, sinusoidal grating fringe patterns with different frequencies in the horizontal and vertical directions are projected on the calibration plate respectively. The sinusoidal grating fringe patterns adopt three different wavelengths, and the wavelengths are: λ ₁ , λ ₂ , λ ₃ , the corresponding main phase values are: φ ₁ , φ ₂ , φ ₃ ; ten images are collected for each projection pattern for ten-step phase-shift code calculation;

Calculate the absolute phase value using the three-frequency heterodyne method:

Superimpose the first phase φ ₁ of the first wavelength λ ₁ and the second phase φ ₂ of the second wavelength λ ₂ to obtain the phase φ ₁₂ , where the wavelength corresponding to the phase φ ₁₂ is λ ₁₂ , and the calculated λ ₁₂ obtained by superimposing for:

Superimpose the second phase φ ₂ of the second wavelength λ ₂ and the third phase φ ₃ of the third wavelength λ ₃ to obtain the phase φ ₂₃ , where the wavelength corresponding to the phase φ ₂₃ is λ ₂₃ , and the calculated λ ₂₃ obtained by superimposing for:

λ ₁₂ and λ ₂₃ are superimposed to obtain the final superimposed wavelength λ ₁₂₃ .

The absolute phase value of the superposition is obtained by the following formula:

Δn _i ∈ [0, 1), i=1, 2, 3, 12, 23, 123,

In the formula,

n _i is the fringe series of a certain point on the surface of the measured object in the corresponding grating diagram, and n _i includes the integer part N _i and the fractional part Δn _i ;

φ _i is the wrapping phase of the corresponding grating;

Φ _i is the absolute phase of the corresponding grating;

The grating 123 is generated by the superposition of the grating 12 and the grating 23, λ ₁ , λ ₂ , λ ₃ are selected so that the wavelength λ _{123 of the grating 123} covers the whole field, and N ₁₂₃ =0, and the absolute phase of the grating 123 is obtained.

As an optional implementation manner, the extracting the center positions of all circular targets in the calibration plate image may include the following steps:

Binarize the image of the calibration plate without grating stripes collected by the camera;

Perform blob analysis on the image of the calibration plate after binarization to obtain the position of the center of mass of the calibration plate circle, and realize the rough positioning of the circular target;

Through Canny edge extraction, the pixel edge position of the circular target is obtained;

By performing grayscale interpolation on the edge pixels of the circular target, and then performing parameter fitting, the sub-pixel edge contour of the circle is obtained;

Least squares ellipse fitting is performed on the obtained sub-pixel edge contour points of each circle to obtain the center position of each ellipse, and the center position of each ellipse is taken as the circle center position of each circular target.

As an optional implementation, sorting all circle centers may include the following steps:

According to the obtained center positions of each ellipse of the calibration plate, determine the four vertices of the irregular quadrilateral in the collected calibration plate pattern;

Connect the four vertices in order to form a quadrilateral;

When the rotation angle of the quadrilateral does not exceed the preset value, determine the position coordinates of each circle point, and determine the order of each circle center according to the order of the position coordinates of each circle point;

When the rotation angle of the quadrilateral exceeds the preset value, the sorting is performed as follows:

Determine all the dots on each side of the quadrilateral, and connect the two opposite dots on the two opposite sides;

Determine all the dots inside each quadrilateral formed by all the connections, and determine the order of the dots.

As an optional implementation manner, when the rotation angle of the quadrilateral does not exceed a preset value, determining the order of the center of each circle may include the following steps:

Determine the position coordinates of the five circular targets in the calibration plate image;

From these large dots with known array coordinates, according to the invariance of linear projection of photographic geometry, the coordinates of small dots located in other positions of the array on the image are estimated;

The coordinates of each circular target are obtained by blob analysis, and the distance from each point to all the estimated points is calculated. The point with the smallest distance is the point of the corresponding position, and it is sorted according to the coordinate order.

As an optional embodiment, determining the position coordinates of the five circular targets in the calibration plate image may include the following steps:

Five circular targets with equal radii are selected, and the radii of the five circular targets with equal radii are larger than the radii of other circular targets in the calibration plate pattern;

Calculate the distance between the points, and get the two points with the largest distance, F1 and F2, and the two points with the smallest distance, N1 and N2, respectively, and the remaining point is determined as one of the five circular targets. The fifth largest dot C5;

Calculate the sum of the distances from F1 and F2 to N1 and N2 respectively;

The distance and the smaller point are the first large circle point C1 in the five circular targets, and the first large circle point is located at the intersection of the horizontal centerline of the pattern and the vertical centerline on the left side of the vertical centerline;

The distance and the larger point are the second largest point C2 in the five circular targets, and the second largest point is located at the intersection of the horizontal centerline of the pattern and the vertical centerline on the right side of the vertical centerline;

Calculate the distances between N1, N2 and C1. The point with the smaller distance is the third largest point C3 in the five circular targets, and the larger point is the fourth largest point C4 in the five circular targets.

In terms of size, the radii of the five great circles are equal, and slightly larger than the radii of the remaining circles of the calibration plate; in terms of arrangement, the two circles with the largest distance belong to the same line as the center circle of the calibration plate pattern, and are located on the left and the left of the middle circle of the pattern respectively. The right side is separated from the center circle by two small circles. The two circles with the smallest distance are in the same row and adjacent to each other. One of the circles belongs to the same column as the center circle of the pattern and is separated by a small circle. When the two circles are located in the pattern When the horizontal center line is below, one of the large circles is on the right side of the vertical center line, and the last large circle also belongs to the same column as the pattern center circle, on the other side of the pattern horizontal center line, and is also separated by a small circle.

During the calibration process, the pose of the calibration board is randomly placed, and the positions of the five great circles are determined, and the direction vectors along the horizontal and vertical directions of the pattern can be obtained, so as to obtain the approximate positions of all points in the pattern. The ellipse is fitted to the extracted real center, and compared with the estimated center distance, the center with the smallest distance is the circle corresponding to the position.

As an optional implementation manner, estimating the coordinates of the small dots may include the following steps:

Determine a horizontal line with two known great circle points C1 and C2, calculate the distance between the two known points in the second line parallel to the line, and the X and Y of the line connecting the two known points direction component, two components in the horizontal direction of the calibration plate are obtained;

Use two known great circle points C3 and C4 to determine a vertical line, the distance between the two known points in the second column parallel to the line, and the X and Y directions of the line connecting the two known points components, two components in the vertical direction of the calibration plate are obtained;

According to the coordinates of the known large circle point C1 and the two components in the horizontal and vertical directions of the calibration plate, the coordinates of all circular targets are estimated, and the position coordinates of all circular targets are stored.

As an optional implementation manner, using a bilinear interpolation algorithm to obtain the absolute phase values of each circle center coordinate in the horizontal and vertical directions may include the following steps:

According to the extracted sub-pixel position coordinates of the center of each circle, the main phase value corresponding to the center of the circle is obtained by bilinear interpolation. The interpolation formula is as follows:

φ(i+u,j+v)=(1-u)(1-v)φ(i,j)+(1-u)vφ(i,j+1)+u(1-v)φ( i+1,j)+uvφ(i+1,j+1);

in,

φ is the main value of the phase;

i, j are the integer parts of the calculated circle center coordinates in the vertical and horizontal directions in the image coordinate system;

u, v are the fractional parts of the calculated circle center coordinates in the vertical and horizontal directions in the image coordinate system;

The phase principal value is obtained as follows:

In the acquired image with the sinusoidal stripe code projected, the light intensity distribution function of the multi-step phase-shift grating is:

I _k (x,y)=I′(x,y)+I″(x,y)cos(φ+(k-1)π/2)

in,

The value of k is 1-n, where n is the number of steps in the multi-step phase shift method;

x, y are pixel coordinates;

I(x, y) is the gray value of the pixel at the position (x, y) in the k-th phase-shift image;

I'(x,y) is the average gray level of the image;

I″(x,y) is the grayscale modulation of the image;

φ is the main value of the phase;

Through the light intensity distribution function, the phase principal value corresponding to each pixel is obtained:

in,

The value of k is 1-n, where n is the number of steps in the multi-step phase shift method;

x, y are pixel coordinates;

I _k is the gray value of the pixel;

N is the number of grating fringe periods;

φ(x,y) is the main value of the phase.

When the ten-step phase-shift method is adopted, in the collected image with the sinusoidal stripe code projected, the light-intensity distribution function of the ten-step phase-shift method grating is:

I _k (x,y)=I′(x,y)+I″(x,y)cos(φ+(k-1)π/2)

in,

The value of k is 1-10;

x, y are pixel coordinates;

I(x, y) is the gray value of the pixel at the position (x, y) in the k-th phase-shift image;

I'(x,y) is the average gray level of the image;

I″(x,y) is the grayscale modulation of the image;

φ is the main value of the phase;

Through the light intensity distribution function, the phase principal value corresponding to each pixel is obtained:

in,

The value of k is 1-10;

x, y are pixel coordinates;

I _k is the pixel gray value of this point;

N is the number of grating fringe periods;

φ(x,y) is the main value of the phase.

The pixel point corresponding to the phase principal value obtained by the above method is an integer coordinate. According to the obtained sub-pixel position coordinates of each circle center, the phase principal value corresponding to the circle center is obtained by bilinear interpolation. The interpolation formula is as follows:

φ(i+u,j+v)=(1-u)(1-v)φ(i,j)+(1-u)vφ(i,j+1)+u(1-v)φ( i+1,j)+uvφ(i+1,j+1)

in,

φ is the main value of the phase;

i, j are the coordinates in the vertical and horizontal directions of the point in the camera image used for interpolation;

u, v are the coordinates in the vertical and horizontal directions of the point in the projector image used for interpolation;

As an optional implementation manner, using the obtained absolute phase values of the coordinates of each circle center in the horizontal and vertical directions, the coordinates in the projector image corresponding to the coordinates of each circle center in the camera image are obtained by the following formula:

in,

u _p is the coordinate of the center point c in the u direction in the projector image;

v _p is the coordinate of the center point c in the v direction in the projector image;

N is the number of grating fringe periods;

W is the resolution of the projector in the horizontal direction;

H is the resolution of the projector in the vertical direction;

Φ _u (u _c ,v _c ) is the absolute phase value in the vertical direction of the center point c;

Φ _v (u _c ,v _c ) is the absolute phase value in the horizontal direction at the center point c;

As an optional implementation manner, the method for calibrating the 3D structured light system further includes: establishing the camera and projector according to the camera and projector internal parameters, distortion coefficients and external parameters, and the coordinate system transformation matrix between the camera and the projector. The mapping relationship between the projector and the world coordinate system can include the following steps:

Through the multi-frequency heterodyne method, the absolute phase grayscale image is obtained; here, the three-frequency heterodyne method can be used;

For any point p[x _w , y _w , z _w ], its projected image coordinates in the camera are p(u _c , _vc ), according to the obtained absolute phase value of each pixel on the captured image, A straight line in the corresponding projector image is obtained, and the corresponding relationship between the camera image and the projector image is:

in:

N is the number of grating fringe periods;

W is the resolution of the projector in the horizontal direction;

Φ(u _c ,v _c ) is the absolute phase value of the pixel;

u _p is the coordinate in the vertical direction in the projector image corresponding to the pixel.

According to the corresponding relationship between the camera image and the projector image, the three-dimensional coordinates (X _W , Y _W , Z _W ) of the unique point p are obtained by the following imaging principle formula:

S _C [u _c , _vc , 1] = A _C M _C [X _W , Y _W , Z _W , 1]

S _P [up , v _p , 1] _{= A P M P [ X W} , Y _W , Z _W , 1]

in,

A _C and A _P are the internal parameters of the camera and projector, respectively;

M _C and M _P are the external parameters of the camera and the projector, respectively;

S _C and S _P are the scale factors of the camera and projector, respectively;

(u _c , _vc ) and ( _{up , v p )} are the image coordinates of the camera and the projector, respectively, both of which use the pre-calibrated system distortion parameters for distortion correction;

(X _W , Y _W , Z _W ) are the unique three-dimensional coordinates of point p.

It can be known from the correspondence expression between the camera and projector images that in the case of projecting horizontal or vertical unidirectional stripes, a pixel coordinate point in the camera corresponds to a coordinate in a horizontal or vertical direction in the projector image. A vertical corresponding line can be determined in the projector image from the vertical absolute phase value, that is, the coordinate value of the corresponding point of the projector image in the vertical direction is determined as the coordinate of the point column; similarly, the horizontal absolute phase value can determine a horizontal line. The corresponding line of , that is, the coordinate value of the corresponding point of the projector image in the horizontal direction is determined as the row coordinate of the point.

Example 1

According to a specific embodiment of the present application, the method for sorting the circle centers in the present application will be described in detail below. The circular target pattern on the calibration plate is shown in Figure 3. Dot sorting includes the following steps:

First determine the position coordinates of the five central circular targets:

1. Calculate the distance between each point, find the two points with the largest distance (set as F1 and F2) and the two points with the smallest distance (set as N1 and N2), then the remaining point can be determined as point 72 (As shown in Figure 3);

2. Calculate the sum of the distances from F1 and F2 to N1 and N2 respectively, then the smaller point is the 47th point, and the larger point is the 53rd point.

3. Calculate the distance between N1, N2 and the 47th point, the point with the smaller distance is the 27th point, and the larger point is the 28th point.

Then, from these large dots with known array coordinates, according to the linear projection invariance of the photographic geometry, the points on the image of the small dots located in other positions of the array are estimated. The main steps are as follows:

1. Determine a straight line with the two known points of the 47th point and the 53rd point, calculate the distance between the two known points in the second row, and the X and Y direction components of the line connecting the two points, so as to obtain the calibration two components in the horizontal direction of the plate;

2. In the same way, the two components in the vertical direction are determined by the 28th point and the 72nd point;

3. From the known coordinates of the 47th point and the respective two components of the horizontal and vertical directions of the calibration plate, the coordinates of the first point are pre-judged, so that the coordinates of the first point and the horizontal and vertical directions of the calibration plate are obtained. The respective two components of , and the coordinates of all 99 points are pre-judged (the coordinates of the great circle have been preferentially extracted before, and their coordinate positions can be stored directly without pre-judgment);

4. The centroid coordinates of all points obtained by blob analysis, calculate the distance from each point to the pre-judged 99 points, and the point with the smallest distance is the point of the corresponding position, so as to realize the sorting and obtain the sorting effect as shown in the figure.

Example 2

According to a specific embodiment of the present application, the specific implementation of the calibration step in the present application will be described in detail below.

1. Obtain the sub-pixel coordinates of the center of each circle through image processing without grating stripes;

2. For the horizontal and vertical grating images, obtain the phase main value of each pixel in the horizontal and vertical directions by ten-step phase shift and multi-frequency heterodyne method;

3. The absolute phase values of the horizontal and vertical directions of the point are obtained by bilinear interpolation;

4. Obtain the projector image coordinates of the point through the correspondence formula between the camera image and the projector image;

5. The pinhole model of the known camera:

In the calibration, the three-dimensional world coordinates in space are (X _W , Y _W , Z _W ), that is, the physical coordinates of the calibration board, (u, v) are the camera image coordinates or projector image coordinates of the extracted center of the circle, and m _ij is Elements of the correspondence matrix.

In the calibration plate pattern, the center-to-center distance is the same, and a plane coordinate system can be established. The obtained coordinates of the center of the circle of each group of camera images and the coordinates of the center of the projector image, as well as the physical coordinates of the calibration plate, can be obtained by Zhang Zhengyou's calibration algorithm. The internal parameters and distortion coefficients of the camera and projector. At the same time, for the calibration board in each pose, Zhang Zhengyou’s calibration method can obtain a set of external parameters for the corresponding camera and projector. Using the internal parameters, distortion coefficients and a set of external parameters can be Determine a set of homography matrices, namely the above M matrices.

Using the obtained extrinsic parameter matrix of the camera in each pose, multiplied by the inverse of the projector extrinsic parameter matrix under the same pose, a new matrix can be obtained, which is the relationship matrix between the camera and the projector. The relationship matrix under one pose, and then the elements at the (i, j) position in the obtained relationship matrix are summed and averaged as the transformation matrix of the camera and the projector.

6. Taking point 47 in Figure 3 as an example, the pinhole model in step 5 contains three equations, and after sorting out Z _C , two linear equations about m _ij can be obtained, namely:

m ₁₁ X _W +m ₁₂ Y _W +m ₁₃ Z _W +m ₁₄ -uX _W m ₃₁ -uY _W m ₃₂ -uZ _W m ₃₃ =um ₃₄

m ₂₁ X _W +m ₂₂ Y _W +m ₂₃ Z _W +m ₂₄ -vX _W m ₃₁ -vY _W m ₃₂ -vZ _W m ₃₃ =vm ₃₄

The camera and projector homography matrices obtained by Zhang Zhengyou's calibration method, respectively, can obtain four linear equations. Since there are only three variables of _XW , _YW , and _ZW , the three-dimensional coordinates of the point can be obtained by simultaneous equations .

Because the extrinsic parameter and transformation relationship matrix only contains rigid relationship of rotation and translation, it will not change the shape and size of the object. Through the external parameter matrix, the three-dimensional coordinates of the space can be converted into the camera coordinate system, and the position coordinates of a point in the camera coordinate system under the projector can be obtained by using the conversion relationship matrix between the camera and the projector.

FIG. 4 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment. As shown in FIG. 4 , the electronic device includes: one or more processors 110 and a memory 120 . A processor 110 is taken as an example in FIG. 4 .

The electronic device may further include: an input device 130 and an output device 140 .

The processor 110 , the memory 120 , the input device 130 and the output device 140 in the electronic device may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 4 .

As a computer-readable storage medium, the memory 120 can be configured to store software programs, computer-executable programs, and modules. The processor 110 executes various functional applications and data processing by running the software programs, instructions and modules stored in the memory 120 to implement any one of the methods in the foregoing embodiments.

The memory 120 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the electronic device, and the like. In addition, the memory may include volatile memory such as random access memory (Random Access Memory, RAM), and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices.

Memory 120 may be a non-transitory computer storage medium or a transitory computer storage medium. The non-transitory computer storage medium, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 120 may optionally include memory located remotely from processor 110, which may be connected to the electronic device via a network. Examples of such networks may include the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 130 may be configured to receive input numerical or character information, and to generate key signal input related to user settings and function control of the electronic device. The output device 140 may include a display device such as a display screen.

This embodiment further provides a computer-readable storage medium storing computer-executable instructions, where the computer-executable instructions are used to execute the above method.

All or part of the processes in the methods of the above-mentioned embodiments can be completed by executing the relevant hardware through a computer program, and the program can be stored in a non-transitory computer-readable storage medium. , wherein the non-transitory computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a RAM, or the like.

Compared with the related art, the present application has the following advantages:

(1) The application adopts a circular target calibration plate, and performs ellipse fitting by extracting the sub-pixel contour of the circle to obtain the sub-pixel position of the center of the circle, and the accuracy is higher. At the same time, during the calibration process, the calibration plate is placed at random, resulting in inconsistent illumination brightness, The target feature of the circle is relatively obvious, and it can also adapt to changes in illumination and accurately extract the position of the center of the circle;

(2) The present application uses bilinear interpolation to obtain the absolute phase value corresponding to each circle center coordinate, and then uses the absolute phase value to calculate the corresponding projector image coordinate.

Claims (12)

1. A method for calibrating a 3D structured light system, comprising:

   make a calibration board;

   acquiring an image of the calibration plate; and

   calibrate the structured light system according to the collected images of the calibration plate;

   Wherein, the step of making a calibration board includes: making a system calibration board and burning it, the pattern on the calibration board is a plurality of circular targets, the center distances of the adjacent circular targets are the same, and a plurality of the circular targets The targets are arranged in rows and columns, and the circular targets are of different sizes;

   Wherein, the step of collecting the calibration plate image includes:

   Set a calibration plate attitude, project sinusoidal grating fringe patterns of multiple frequencies in the horizontal and vertical directions on the calibration plate respectively, project light without grating fringes on the other calibration plate, and place it on the calibration plate. At the same time of each projection, the camera is used to collect the images of the calibration plate projected with various patterns in the posture; and

   Adjust the posture of the calibration plate, repeat the steps of collecting the images of the calibration plate, until the number of postures of the calibration plate reaches a preset target value, and obtain a plurality of images of the calibration plate;

   The step of calibrating the structured light system includes:

   extracting the center positions of all the circular targets in the calibration plate image;

   Sort all circle centers according to the extracted center positions of all the circular targets;

   The absolute phase value of each circle center coordinate is obtained by bilinear interpolation algorithm;

   Using the obtained absolute phase value of the coordinates of each circle center, the coordinates in the projector image corresponding to the coordinates of each circle center in the camera image are obtained;

   According to the obtained coordinates of the circle center in the camera and the projector image, the internal parameters, distortion coefficients and external parameters of the camera and the projector are obtained by Zhang Zhengyou's calibration method;

   Utilize the external parameters of the camera and the projector under the same described calibration plate attitude, calculate the transformation matrix between the camera and the projector coordinate system under each calibration plate attitude; And

   According to the obtained transformation matrix under the attitude of each calibration board, the calibrated camera and projector coordinate system transformation matrix is obtained.
2. The method for calibrating a 3D structured light system according to claim 1, before the bilinear interpolation algorithm is used to obtain the absolute phase value of each circle center coordinate, the method for calibrating the 3D structured light system further comprises: using multi-frequency and multi-step The phase shift algorithm is used to obtain the absolute phase values of the plane phase field of the calibration plate in the horizontal and vertical directions.
3. The method for calibrating a 3D structured light system according to claim 1, wherein the step of extracting the center positions of all the circular targets in the calibration plate image comprises:

   Binarize the image of the calibration plate without grating stripes collected by the camera;

   Perform blob analysis on the image of the calibration plate after binarization to obtain the position of the center of mass of the calibration plate circle, and realize the rough positioning of the circular target;

   Through Canny edge extraction, the pixel edge position of the circular target is obtained;

   By performing grayscale interpolation on the edge pixels of the circular target and then performing parameter fitting, the sub-pixel edge contour of the circle is obtained; and

   Perform least squares ellipse fitting on the obtained sub-pixel edge contour points of each circle to obtain the center position of each ellipse, and use the center position of each ellipse as the circle center position of each circular target.
4. The method for calibrating a 3D structured light system according to claim 3, wherein the step of sorting all circle centers comprises:

   According to the obtained center position of each ellipse of the calibration plate, determine the four vertices of the irregular quadrilateral in the collected calibration plate pattern;

   Wire the four vertices in sequence into a quadrilateral; and

   When the angle of rotation of the quadrilateral does not exceed the preset value, the position coordinates of each circle point are determined, and the order of each circle center is determined according to the order of the position coordinates of each circle point;

   Wherein, when the rotation angle of the quadrilateral exceeds the preset value, the sorting is performed according to the following method:

   determining all the dots on each side of the quadrilateral, and connecting the two opposite dots on the two opposite sides; and

   Identify all the dots inside each quadrilateral formed by all the lines, and determine the order of each dot.
5. The method for calibrating a 3D structured light system according to claim 4, wherein, when the rotation angle of the quadrilateral does not exceed a preset value, determining the order of each circle center comprises the following steps:

   Determine the position coordinates of the five circular targets in the calibration plate image;

   From these large dots with known array coordinates, according to the linear projection invariance of photographic geometry, estimate the coordinates of small dots located elsewhere in the array on the image; and

   The coordinates of each circular target are obtained by blob analysis, and the distance from each point to all the estimated points is calculated. The point with the smallest distance is the point of the corresponding position, which is sorted according to the coordinate order.
6. The method for calibrating a 3D structured light system according to claim 5, wherein determining the position coordinates of the five circular targets in the calibration plate image comprises:

   Five circular targets with equal radii are selected, and the radii of the five circular targets with equal radii are larger than the radii of other circular targets in the calibration plate pattern;

   Calculate the distance between each point, and get the two points with the largest distance, F1 and F2, and the two points with the smallest distance, N1 and N2, respectively. The remaining point is determined as one of the five circular targets. The fifth largest dot C5;

   Calculate the sum of the distances from F1 and F2 to N1 and N2 respectively;

   The distance and the smaller point are the first large circle point C1 in the five circular targets, and the first large circle point is located at the intersection of the horizontal centerline of the pattern and the vertical centerline on the left side of the vertical centerline;

   The distance and the larger point is the second largest point C2 in the five circular targets, and the second largest point is located at the intersection of the horizontal centerline of the pattern and the vertical centerline to the right of the vertical centerline; and

   Calculate the distances between N1, N2 and C1. The point with the smaller distance is the third largest point C3 in the five circular targets, and the larger point is the fourth largest point C4 in the five circular targets.
7. The method for calibrating a 3D structured light system according to claim 5, wherein estimating the coordinates of the small dots comprises:

   Determine a horizontal line with two known great circle points C1 and C2, calculate the distance between the two known points in the second line parallel to the line, and the X and Y of the line connecting the two known points direction component, two components in the horizontal direction of the calibration plate are obtained;

   Use two known great circle points C3 and C4 to determine a vertical line, the distance between the two known points in the second column parallel to the line, and the X and Y directions of the line connecting the two known points components, resulting in two components in the vertical direction of the calibration plate; and

   According to the known coordinates of the large circle point C1 and the respective two components of the horizontal and vertical directions of the calibration plate, the coordinates of all circular targets are estimated and obtained, and the position coordinates of all the circular targets are stored.
8. The method for calibrating a 3D structured light system according to claim 3, wherein the step of obtaining the absolute phase value of each circle center coordinate in the horizontal and vertical directions by using a bilinear interpolation algorithm comprises:

   According to the obtained sub-pixel position coordinates of each circle center, the main phase value corresponding to the circle center is obtained by bilinear interpolation. The interpolation formula is as follows:

   φ(i+u,j+v)=(1-u)(1-v)φ(i,j)+(1-u)vφ(i,j+1)+u(1-v)φ( i+1,j)+uvφ(i+1,j+1);

   in,

   φ is the main value of the phase;

   i, j are the integer parts of the calculated circle center coordinates in the vertical and horizontal directions in the image coordinate system;

   u, v are respectively the fractional parts of the calculated circle center coordinates in the vertical and horizontal directions in the image coordinate system;

   The phase principal value is obtained as follows:

   In the acquired image with the sinusoidal stripe code projected, the light intensity distribution function of the multi-step phase-shift grating is:

   I _k (x,y)=I′(x,y)+I″(x,y)cos(φ+(k-1)π/2)

   in,

   The value of k is 1-n, where n is the number of steps in the multi-step phase shift method;

   x, y are pixel coordinates;

   I(x, y) is the gray value of the pixel at the position (x, y) in the k-th phase-shift image;

   I'(x,y) is the average gray level of the image;

   I"(x,y) is the grayscale modulation of the image;

   φ is the main value of the phase;

   Through the light intensity distribution function, the phase principal value corresponding to each pixel is obtained:

   

   in,

   The value of k is 1-n, where n is the number of steps in the multi-step phase shift method;

   x, y are pixel coordinates;

   I _k is the gray value of the pixel;

   N is the number of grating fringe periods;

   φ(x,y) is the main value of the phase.
9. The method for calibrating a 3D structured light system according to claim 8, the punctuation method for the 3D structured light system further comprises: using the obtained absolute phase values of the coordinates of each circle center in the horizontal and vertical directions to obtain each The coordinates of the center of the circle in the camera image correspond to the coordinates in the projector image:

   

   

   in,

   u _p is the coordinate of the center point c in the u direction in the projector image;

   v _p is the coordinate of the center point c in the v direction in the projector image;

   N is the number of grating fringe periods;

   W is the resolution of the projector in the horizontal direction;

   H is the resolution of the projector in the vertical direction;

   Φ _u (u _c ,v _c ) is the absolute phase value in the vertical direction of the center point c;

   Φ _v (u _c , v _c ) is the absolute phase value in the horizontal direction at the center point c.
10. The method for calibrating a 3D structured light system according to claim 1, the method for calibrating the 3D structured light system further comprises: according to the internal parameters, distortion coefficients and external parameters of the camera and the projector obtained from the calibration, and the difference between the camera and the projector The coordinate system transformation matrix of , establishes the mapping relationship between the camera and the projector and the world coordinate system;

    The steps of establishing the mapping relationship between the camera and the projector and the world coordinate system include:

    Through the multi-frequency heterodyne method, the absolute phase grayscale image is obtained;

    For any point p[x _w , y _w , z _w ], the image coordinate of this point projected in the camera is p(u _c , _vc ), according to the obtained absolute phase value of each pixel on the captured image , a straight line in the corresponding projector image is obtained, and the corresponding relationship between the camera image and the projector image is:

    

    in:

    N is the number of grating fringe periods;

    W is the resolution of the projector in the horizontal direction;

    Φ(u _c ,v _c ) is the absolute phase value of the pixel;

    u _p is the vertical coordinate in the projector image corresponding to the pixel; and

    According to the corresponding relationship between the camera image and the projector image, the three-dimensional coordinates (X _W , Y _W , Z _W ) of the unique point p are obtained by the following imaging principle formula:

    S _C [u _c , _vc , 1] = A _C M _C [X _W , Y _W , Z _W , 1]

    S _P [up , v _p , 1] _{= A P M P [ X W} , Y _W , Z _W , 1]

    in,

    A _C and A _P are the internal parameters of the camera and projector, respectively;

    M _C and M _P are the external parameters of the camera and the projector, respectively;

    S _C and S _P are the scale factors of the camera and projector, respectively;

    (u _c , _vc ) and ( _{up , v p )} are the image coordinates of the camera and the projector, respectively, both of which use the pre-calibrated system distortion parameters for distortion correction;

    (X _W , Y _W , Z _W ) are the unique three-dimensional coordinates of point p.
11. An electronic device comprising:

    processor;

    memory, set to store programs,

    When the program is executed by the processor, the processor implements the method for calibrating a 3D structured light system according to any one of claims 1-10.
12. A computer-readable storage medium storing computer-executable instructions for executing the method for calibrating a 3D structured light system according to any one of claims 1-10.

Images