CN108225216B - Structured light system calibration method and device, structured light system and mobile device - Google Patents
Structured light system calibration method and device, structured light system and mobile device Download PDFInfo
- Publication number
- CN108225216B CN108225216B CN201611150961.6A CN201611150961A CN108225216B CN 108225216 B CN108225216 B CN 108225216B CN 201611150961 A CN201611150961 A CN 201611150961A CN 108225216 B CN108225216 B CN 108225216B
- Authority
- CN
- China
- Prior art keywords
- image
- structured light
- projection
- light system
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
- G01B11/2531—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object using several gratings, projected with variable angle of incidence on the object, and one detection device
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention discloses a method and a device for calibrating a structured light system, the structured light system and mobile equipment, wherein the method for calibrating the structured light system comprises the following steps: obtaining a calibration plate image and a projection image of the calibration plate projected by a projection module, wherein the projection pattern of the projection module is formed based on a binary geometric space coding pattern; and calibrating the structured light system comprising the camera and the projection module by utilizing the calibration plate image and the projection image. The invention can improve the calibration precision of the structured light system. When the camera adopts an infrared camera and the projection module adopts a projection grating module, and the structured light system is a grating structured light system, the system can also provide a structured light system with high reliability, small volume and low cost for the mobile equipment, thereby realizing high-precision three-dimensional scanning of the mobile equipment.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a method and a device for calibrating a structured light system, the structured light system and mobile equipment.
Background
The structured light three-dimensional scanning technology has the advantages of low cost, non-contact, high precision, high efficiency and the like, is widely applied to industries such as product design and manufacture, industrial measurement, quality detection, medicine, video entertainment and the like, and is praised as the most promising three-dimensional measurement method. The structured light three-dimensional scanning technology realizes three-dimensional scanning by actively controlling the light source, and has higher reliability compared with other three-dimensional scanning technologies. Before the structured light three-dimensional reconstruction is carried out, the structured light system is calibrated, namely, the internal parameters of a camera and projection equipment in the structured light system and the conversion relation between the camera and the projection equipment are obtained, and the internal parameters are also called as external parameters. The quality of the three-dimensional reconstruction is directly influenced by the calibration precision. At present, the calibration technology for cameras is mature, a chessboard calibration board can be used for calibrating the cameras, and the projection equipment is non-imaging equipment, so that the corresponding relation between a three-dimensional space point and the corresponding position of the three-dimensional space point on a projection image is difficult to accurately acquire, and the calibration of the projection equipment is relatively difficult.
Disclosure of Invention
The embodiment of the invention provides a structured light system calibration method, which is used for improving the calibration precision of a structured light system and comprises the following steps:
obtaining a calibration plate image and a projection image of the calibration plate projected by a projection module, wherein the projection pattern of the projection module is formed based on a binary geometric space coding pattern;
and calibrating the structured light system comprising the camera and the projection module by utilizing the calibration plate image and the projection image.
In one embodiment, the camera is an infrared camera; the projection module is a projection grating module; the structured light system is a grating structured light system.
The embodiment of the invention also provides a calibration device of the structured light system, which is used for improving the calibration precision of the structured light system and comprises the following components:
the image obtaining module is used for obtaining a calibration plate image and a projected image of the calibration plate projected by the projection module, wherein the projection pattern of the projection module is formed on the basis of a binary geometric space coding pattern;
and the system calibration module is used for calibrating the structured light system comprising the camera and the projection module by utilizing the calibration plate image and the projection image.
In one embodiment, the camera is an infrared camera; the projection module is a projection grating module; the structured light system is a grating structured light system.
In the method and the device for calibrating the structured light system, the projection pattern of the projection module is formed based on the binary geometric space coding pattern, and the binary geometric space coding pattern has the characteristics of strong robustness, high coding density and small window, so that the structured light system is calibrated by utilizing the projection image projected by the projection module on the calibration plate, and the calibration precision of the structured light system can be improved; moreover, the calibration structured light system is based on the calibration plate image and the projection image, no camera calibration error is introduced in the calibration process, and the calibration precision of the structured light system can be improved. Furthermore, in the embodiment, when the infrared camera is used as the camera, the projection module is a projection grating module, and the structured light system is a grating structured light system, high-precision calibration of the grating structured light system with high reliability, small volume and low cost can be realized.
The embodiment of the invention also provides a structured light system, which is used for improving the calibration precision of the structured light system and comprises the following components:
the camera is used for shooting the image of the calibration plate and the projected image of the calibration plate projected by the projection module;
the projection module is used for projecting a projection pattern formed based on the binary geometric space coding pattern to the calibration plate;
the calibration plate image and the projection image are used for calibrating a structured light system comprising a camera and a projection module.
In one embodiment, the camera is an infrared camera; the projection module is a projection grating module; the structured light system is a grating structured light system.
The structured light system provided by the embodiment of the invention can provide a calibration plate image and a projected image of the calibration plate projected by the projection module, and is used for calibrating the structured light system, wherein the projection pattern of the projection module is formed based on a binary geometric space coding pattern, and the binary geometric space coding pattern has the characteristics of strong robustness, high coding density and small window, so that the calibration precision of the structured light system is favorably improved. Furthermore, in the embodiment, when the infrared camera is used as the camera, the projection module is a projection grating module, and the structured light system is a grating structured light system, the structured light system with high reliability, small volume and low cost can be provided for the mobile device, and the requirement of high-precision three-dimensional scanning of the mobile device is met.
An embodiment of the present invention further provides a mobile device, configured to improve calibration accuracy of a structured light system, where the mobile device includes: the structured light system calibration device and the structured light system are provided.
The mobile device of the embodiment of the invention can realize high-precision three-dimensional scanning of the mobile device by using the structured light system and the structured light system calibration device, and further, in the embodiment, when the camera adopts an infrared camera, the projection module adopts a projection grating module, and the structured light system is a grating structured light system, the structured light system with high reliability, small volume and low cost can be provided for the mobile device, so that the high-precision three-dimensional scanning of the mobile device is realized.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. In the drawings:
FIG. 1 is a diagram illustrating a method for calibrating a structured light system according to an embodiment of the present invention;
FIG. 2 is an exemplary diagram of a projected pattern in an embodiment of the present invention;
FIG. 3 is a diagram illustrating a calibration apparatus of a structured light system according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a structured light system in accordance with an embodiment of the present invention;
FIG. 5 is a diagram of a mobile device in an embodiment of the invention;
FIG. 6 is a diagram showing an example of a standard flat panel of an experimental subject according to an embodiment of the present invention;
FIG. 7 is a diagram illustrating an exemplary result of feature point detection obtained by an experiment in the embodiment of the present invention;
FIG. 8 is an exemplary diagram of a point cloud after three-dimensional scanning obtained by an experiment in an embodiment of the present invention;
fig. 9 is an exemplary diagram of a curved surface reconstructed by using point cloud obtained through experiments in the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
The inventor finds that, in the process of calibrating the structured light system, calibration of the projection device is relatively difficult because the projection device is a non-imaging device, and it is difficult to accurately obtain a corresponding relationship between a three-dimensional space point and a corresponding position of the three-dimensional space point on a projection image. Some existing schemes use camera calibration results to calibrate the projection device, however, camera calibration errors are introduced, and the calibration accuracy of the structured light system is reduced. To address this problem, an embodiment of the present invention provides a method for calibrating a structured light system, as shown in fig. 1, where the method may include:
As can be known from the flow shown in fig. 1, in the embodiment of the present invention, the projection pattern of the projection module is formed based on the binary geometric space coding pattern, which is different from the existing method that only uses the projection squares, dots, or stripes, because the binary geometric space coding pattern has strong robustness, high coding density, and small window characteristics, the projection image projected by the projection module on the calibration board is used to calibrate the structured light system, so as to improve the calibration accuracy of the structured light system. In addition, the calibration structured light system in the embodiment of the invention is based on the calibration plate image and the projection image, and no camera calibration error is introduced in the calibration process, so that the high precision of the calibration of the structured light system can be ensured.
How to calibrate the structured light system in the embodiment of the present invention is described in detail with reference to specific examples.
Firstly, when in implementation, a calibration plate image and a projection image of the calibration plate projected by the projection module are obtained. The calibration plate images and the projection images are shot by the camera, in the embodiment, the camera can shoot a plurality of groups of calibration plate images and projection images, and the structured light system is calibrated by the aid of the plurality of groups of calibration plate images and projection images, so that calibration precision can be further improved. For example, the calibration board can be placed in the field of view of the camera and the projection module, the camera first shoots an image of the calibration board, then the projection module is used for projecting a projection pattern onto the calibration board, then the camera shoots a projection image formed by projection on the calibration board, the relative position of the calibration board and the camera is changed, the camera shoots multiple groups of images of the calibration board and the projection image, the more the number of the shot images is, the higher the calibration precision is, in the embodiment, the shooting of 15-20 groups is preferred, and therefore the calibration precision can be improved without increasing too much calculation amount. It should be noted that, in the following embodiments, the structured light system calibration is implemented by using the zhangyoubi plane calibration method, in this case, at least 3 sets of calibration plate images and projection images need to be captured by the camera, and those skilled in the art will readily understand that fewer calibration plate images and projection images can be captured when other specific calibration methods are used, and the structured light system calibration can also be implemented. The calibration board can adopt a checkerboard calibration board, and can also adopt calibration boards with other patterns.
The projection pattern of the projection module is formed based on a binary geometric space coding pattern. In an embodiment, the binary geometric spatial coding pattern may be generated by embedding a plurality of codeword graphs into a feature graph by using a pseudorandom array, where the codeword graph is a black-and-white binary geometric graph, the feature graph is a grid, lines and points in the codeword graph are white, and a background color is black; the projection pattern may be formed by placing white in the binary geometric space encoding pattern as transparent and black as opaque. The grid as the feature pattern may take various shapes, for example, may be formed by perpendicularly intersecting straight lines, or may be replaced with other patterns. The projection pattern formed based on the binary geometric space coding pattern can be used for realizing high-precision calibration of the structured light system and can also be used for realizing high-precision three-dimensional measurement.
Fig. 2 gives an example of a projected pattern. As shown in fig. 2, in this example, a pseudo random array with a window size of 2 × 2 and coding elements of 8 and a size of 65 × 63 may be generated by using a pseudo random coding principle, 8 black-and-white binary geometric figures with simple structure, different shapes and large difference are designed as codeword graphs, meanwhile, a grid formed by perpendicularly intersecting straight lines is designed as a feature graph, the codeword graphs are embedded in the feature graphs, black is selected as a background color, thereby generating a binary geometric space coding pattern, an intersection point of two perpendicularly intersecting straight lines is positioned as a feature point of the binary geometric space coding pattern, and a corresponding calibration plate adopts a checkerboard calibration plate. As shown in fig. 2, the binary geometric spatial coding pattern has three features: the method has the advantages of high coding density, small window size and strong robustness. Then, the white color in the binary geometric space coding pattern is set as transparent, and the black color is set as opaque, so that a projection pattern can be formed. The 8 black-white binary geometric figures shown in fig. 2 are only an example, and those skilled in the art will readily understand that other figures may be selected according to actual requirements during implementation, and the calibration requirements and the three-dimensional scanning requirements of the structured light system may also be met.
After obtaining the calibration plate image and the projection image, the structured light system comprising the camera and the projection module is calibrated by utilizing the calibration plate image and the projection image. The structured light system is calibrated by utilizing the calibration plate image and the projected image, the corresponding relation between the three-dimensional space point and the corresponding position of the three-dimensional space point on the projected image can be accurately obtained, and a camera calibration error can not be introduced in the calibration process. In a specific embodiment, the coordinates of the projected image on the camera image can be extracted first, and the corresponding relation between the camera image coordinates and the projection module image coordinates is established by combining the coordinates of the projected image on the projection module image; and calibrating the structured light system by using the corresponding relation and the calibration plate image.
Specifically, extracting coordinates of the projected image on the camera image, and establishing a corresponding relationship between the coordinates of the camera image and the coordinates of the projection module image by combining the coordinates of the projected image on the projection module image, may include: converting the projected image into a gray image, and smoothing the gray image by using a filter; extracting characteristic points from the smoothed gray level image to obtain a topological structure of the characteristic points; extracting a code word graph in the grid according to the topological structure of the feature points; identifying the extracted code word graph and determining the code words of the characteristic points; and determining the corresponding relation between the camera image coordinates and the projection module image coordinates according to the code word information of the identified characteristic points and the code word information of the projection characteristic points.
For example, the specific process of establishing the correspondence between the camera image coordinates and the projection module image coordinates may include:
1. and converting the projected image shot by the camera into a gray image, and smoothing the gray image by using a filter. In the embodiment, a cross template is designed to filter the gray image, and the cross template can be expressed by the following formula:
wherein H represents a filtering threshold value, I represents a projection image shot by a camera, w represents the radius of a cross-shaped template, l is w/3, and alpha, beta, I, j, gamma and eta are coordinate variables in the template. Setting a proper threshold value H to extract candidate characteristic points, and eliminating error characteristic points by using the principle of unchanged symmetry after rotating 180 degrees. It should be noted that, in this embodiment, the cross template is designed to filter the grayscale image, and those skilled in the art will readily understand that in the specific implementation, other templates may be selected according to actual requirements to filter the grayscale image.
2. And extracting characteristic points from the smoothed gray level image to obtain a topological structure of the characteristic points, and extracting a code word graph in the grid according to the topological structure of the characteristic points. When the calibration plate adopts a checkerboard calibration plate, because each grid image has 4 feature points, a codeword graph in the grid can be extracted according to the topological structure, and for example, the codeword graph can be normalized into an image with the size of 28 × 28 by using a projective transformation principle and a bilinear interpolation method by means of the 4 feature points forming the grid in which the codeword graph exists.
3. And identifying the extracted code word graph and determining the code words of the characteristic points.
4. And determining the corresponding relation between the camera image coordinates and the projection module image coordinates according to the code word information of the identified characteristic points and the code word information of the projection characteristic points.
In a specific embodiment, since the projection image is easily affected by the color on the calibration board when being projected onto the calibration board, so that the codeword pattern is not easily recognized correctly, the extracted codeword pattern can be recognized by using the convolutional neural network as follows: before identification, a plurality of projection patterns containing single code word patterns are respectively projected to a calibration plate to be used for collecting training samples, the number of the samples is enlarged by adjusting Gaussian noise, executing affine transformation and applying Gaussian filtering, a sample training identification network is utilized, and the trained network is utilized to identify the extracted code word patterns. For example, the convolutional neural network Lenet-5 may be used to identify the codeword patterns, before identification, each projection pattern containing only a single codeword pattern is projected onto a calibration board for collecting training samples, and the number of samples is enlarged by adjusting gaussian noise, performing a slight affine transformation, and performing gaussian filtering, so that the network trained by using the samples has a stronger identification capability. And then, recognizing the extracted codeword graph by using the trained network, and further determining the codewords of the feature points.
After the corresponding relation between the image coordinates of the camera and the image coordinates of the projection module is obtained, the structured light system can be calibrated by using the corresponding relation and the calibration plate image. In a specific example, the coordinate of the calibration plate on the camera image may be converted to the projection module image by using the corresponding relationship, and then the structured light system may be calibrated by using the zhangyoubi plane calibration method. Those skilled in the art will readily understand that the calibration of the structured light system by the Zhangyingyou plane calibration method is only an example, and other calibration methods may be used to calibrate the structured light system in the embodiment.
Specifically, before the structured light system is calibrated by using the Zhang Zhengyou plane calibration method, a homography matrix between the camera image and the projection module image can be calculated by using the corresponding relation between the camera image coordinate and the projection module image coordinate, and the homography matrix is used for converting the characteristic point coordinate of the calibration plate on the camera image to the projection module image. And subsequently, the structured light system can be calibrated by taking the characteristic points on the calibration plate as data sources and combining a Zhangyingyou plane calibration method.
An example of a specific process for calibrating the structured light system is illustrated below, in which a zhangnyou plane calibration method is used to calibrate the structured light system.
In this example, the calibration board adopts a checkerboard calibration board, an angular point detection algorithm is adopted to detect the angular point coordinates on the checkerboard calibration board image, homography matrix between the camera image and the projection module image is utilized to convert the angular point coordinates of the checkerboard calibration board on the camera image to the projection module image, the angular point on the checkerboard calibration board is taken as a data source to calibrate the camera and the projection module simultaneously in combination with Zhang Zhenyou plane calibration method, so as to obtain the internal and external parameters of the camera and the projection module, and the external parameters of the structured light system are calculated by utilizing the external parameters of the camera and the projection module, including rotation matrix RsAnd translation matrix TsThe calculation formula is as follows:
wherein R iscRepresenting the rotation matrix of the camera, RpRotation matrix, T, representing a projection modulecRepresenting the translation matrix, T, of the camerapA translation matrix representing the projection module. Because the external parameters of the structured light system directly calculated by the formula (2) have larger errors, further optimization can be carried out according to a back projection error principle calibrated by a camera:
ec＝mc-g-1(m′p,Kc,θc,Rc,Tc,Kp,θp) (3)
wherein e iscRepresenting a back projection error; m iscIs the actual feature point coordinates of the projected pattern on the camera image; g-1() Representing the transformation of the projection module image into a camera image; m'pThe coordinates of an ideal corner point of the projection pattern on the projection module image; kcAnd thetacRespectively representing calibrated camera intrinsic parameters and distortion coefficients; rcAnd TcAs an external parameter of the camera, take here
Tc＝Ts+RsTp；KpAnd thetapRespectively representing the calibrated internal parameters and distortion coefficients of the projection module; the optimization objective function is obtained from equation (3):
the optimization problem of formula (4) is a non-linear least squares problem, and R can be calculated according to formula (2)sAnd TsAnd solving the initial value by using a Levenberg-Marquart algorithm, so that the internal and external parameters of the optimized camera-projection module structured light system can be obtained.
Based on the same inventive concept, an embodiment of the present invention further provides a device for calibrating a structured light system, as described in the following embodiments. The principle of the structured light system calibration device for solving the problems is similar to that of the structured light system calibration method, so the implementation of the structured light system calibration device can refer to the implementation of the structured light system calibration method, and repeated parts are not described again.
As shown in fig. 3, the device for calibrating a structured light system in the embodiment of the present invention may include:
an image obtaining module 301, configured to obtain a calibration plate image and a projection image of the calibration plate projected by a projection module, where a projection pattern of the projection module is formed based on a binary geometric space coding pattern;
a system calibration module 302 for calibrating the structured light system including the camera and the projection module using the calibration plate image and the projection image.
In a specific embodiment, the binary geometric spatial coding pattern may be generated by embedding a plurality of codeword patterns into a feature pattern by using a pseudo-random array, where the codeword patterns are black-and-white binary geometric patterns, the feature pattern is a grid, and the background color is black; the projection pattern may be formed by placing white in the binary geometric space encoding pattern as transparent and black as opaque.
In a particular embodiment, the grid may be formed by perpendicularly intersecting straight lines.
In a specific embodiment, the system calibration module 302 may be specifically configured to: extracting coordinates of the projected image on the camera image, and establishing a corresponding relation between the camera image coordinates and the projection module image coordinates by combining the coordinates of the projected image on the projection module image; and calibrating the structured light system by utilizing the corresponding relation and the calibration plate image.
In a specific embodiment, the system calibration module 302 may be specifically configured to: converting the projected image into a gray image, and smoothing the gray image by using a filter; extracting characteristic points from the smoothed gray level image to obtain a topological structure of the characteristic points; extracting a code word graph in the grid according to the topological structure of the feature points; identifying the extracted code word graph and determining the code words of the characteristic points; and determining the corresponding relation between the camera image coordinates and the projection module image coordinates according to the code word information of the identified characteristic points and the code word information of the projection characteristic points.
In a specific embodiment, the system calibration module 302 may be specifically configured to identify the extracted codeword graph by using a convolutional neural network as follows: before identification, a plurality of projection patterns containing single code word patterns are respectively projected to a calibration plate to be used for collecting training samples, the number of the samples is enlarged by adjusting Gaussian noise, executing affine transformation and applying Gaussian filtering, a sample training identification network is utilized, and the trained network is utilized to identify the extracted code word patterns.
In a specific embodiment, the system calibration module 302 may be specifically configured to: converting the coordinates of the calibration plate on the camera image to the projection module image by utilizing the corresponding relation between the camera image coordinates and the projection module image coordinates; and calibrating the structured light system by using a Zhangyingyou plane calibration method.
In a specific embodiment, the system calibration module 302 may be specifically configured to: calculating a homography matrix between the camera image and the projection module image by utilizing the corresponding relation between the camera image coordinate and the projection module image coordinate; converting the characteristic point coordinates of the calibration plate on the camera image to the projection module image by using the homography matrix; and (3) calibrating the structured light system by taking the characteristic points on the calibration plate as data sources and combining a Zhangyingyou plane calibration method.
As described above, in the calibration apparatus for a structured light system according to the embodiment of the present invention, the projection pattern of the projection module is formed based on the binary geometric spatial coding pattern, and the binary geometric spatial coding pattern has strong robustness, high coding density, and small window characteristics, so that the structured light system is calibrated by using the projection image projected by the projection module on the calibration board, and the calibration accuracy of the structured light system can be improved; moreover, the calibration structured light system is based on the calibration plate image and the projection image, no camera calibration error is introduced in the calibration process, and the calibration precision of the structured light system can be improved.
Based on the same inventive concept, embodiments of the present invention further provide a structured light system, as described in the following embodiments. The principle of the structured light system for solving the problem is similar to that of the structured light system calibration method, so the implementation of the structured light system can refer to the implementation of the structured light system calibration method, and repeated parts are not described again.
As shown in fig. 4, the structured light system in the embodiment of the present invention may include: the camera 401 is used for shooting the image of the calibration plate and the projected image of the calibration plate projected by the projection module; a projection module 402, configured to project a projection pattern formed based on the binary geometric space encoding pattern onto the calibration board; the calibration plate image and the projection image are used for calibrating a structured light system comprising a camera and a projection module.
The structured light system provided by the embodiment of the invention can provide a calibration plate image and a projected image of the calibration plate projected by the projection module, and is used for calibrating the structured light system, wherein the projection pattern of the projection module is formed based on a binary geometric space coding pattern, and the binary geometric space coding pattern has the characteristics of strong robustness, high coding density and small window, so that the calibration precision of the structured light system is favorably improved.
In a specific embodiment, the binary geometric spatial coding pattern may be generated by embedding a plurality of codeword patterns into a feature pattern by using a pseudo-random array, where the codeword patterns are black-and-white binary geometric patterns, the feature pattern is a grid, and the background color is black; the projection pattern may be formed by placing white in the binary geometric space encoding pattern as transparent and black as opaque.
In a particular embodiment, the grid may be formed by perpendicularly intersecting straight lines.
Further, the inventor finds that, after Microsoft corporation and Google corporation respectively launch the Kinect series and Tango series three-dimensional scanning products, many scientific and technological enterprises around the world now try to add a high-precision three-dimensional scanning function to a mobile device (such as a mobile phone, Pad) to attract the eyes of consumers and increase the sales volume of products. However, there is no solution in the prior art that can add a high-precision three-dimensional scanning function to the mobile device.
Therefore, in the structured light system according to the embodiment of the present invention, the infrared camera is used as the camera, the projection module is a projection grating module, and the structured light system is a grating structured light system, which is a structured light system with high reliability, small volume and low cost designed for the mobile terminal. In a specific embodiment, the grating structured light system may be formed by a camera and a micro structured light projection grating module, and the grating structured light system may be manufactured by a photolithography process to form a projection grating module capable of projecting specific structured light in cooperation with a related micro optical projection module, so that the grating structured light system is miniaturized as a whole, and a structured light system with high reliability, small volume and low cost is provided for a mobile device, thereby realizing high-precision three-dimensional scanning of the mobile device.
From the perspective of the optical model, the projection grating module can be viewed as an "inverse" camera that projects two-dimensional images into a three-dimensional scene, with a basic grating structured light system equivalent to a binocular stereoscopic system. The binocular stereo vision adopts the function of simulating human eyes by two cameras, obtains the surface information of an object from different visual angles, and realizes three-dimensional reconstruction by using parallax. Because a camera in binocular vision is replaced by the projection grating module, a series of characteristic points can be projected to the surface to be measured by means of the projection grating module, and the problem that corresponding point matching of different images in the binocular vision is difficult to realize is solved.
The grating structure optical system adopts a projection grating module, so when the grating structure optical system is calibrated, a photoetching pattern in the projection grating module is needed. The lithographic pattern is the previously described projection pattern formed based on the binary geometric spatial encoding pattern. In a specific embodiment, the binary geometric spatial coding pattern may be generated by embedding a plurality of codeword patterns into a feature pattern by using a pseudo-random array, where the codeword patterns are black-and-white binary geometric patterns, the feature pattern is a grid, and the background color is black; when the projection module adopts a projection grating module, the white photoetching in the binary geometric space coding pattern can be changed into light transmission by utilizing the photoetching process, and the black photoetching in the binary geometric space coding pattern can be changed into light non-transmission by utilizing the photoetching process, so that the photoetching pattern is formed. The projection grating module projects the photoetching pattern to the calibration plate, and the camera shoots a projection image projected by the projection module after the photoetching pattern is projected.
Based on the same inventive concept, the embodiment of the present invention further provides a mobile device, which includes the above-mentioned structured light system calibration apparatus and the above-mentioned structured light system.
Fig. 5 is a schematic structural diagram of a mobile device in an embodiment of the present invention, and as shown in fig. 5, the mobile device in the embodiment of the present invention includes a structured light system 51 and a structured light system calibration device 52, where the structured light system 51 includes a camera 401 and a projection module 402. In one embodiment, camera 401 may be an infrared camera, projection module 402 may be a projection grating module, and structured light system 51 may be a grating structured light system.
According to the foregoing description, it can be seen that the mobile device according to the embodiment of the present invention can implement high-precision three-dimensional scanning of the mobile device by using the structured light system and the structured light system calibration device, and further, in the embodiment, when the camera adopts an infrared camera, the projection module adopts a projection grating module, and the structured light system is a grating structured light system, the structured light system with high reliability, small volume and low cost can be provided for the mobile device, so as to implement high-precision three-dimensional scanning of the mobile device.
As can be understood from the foregoing description of the embodiment, in the calibration method and apparatus of a structured light system according to the embodiments of the present invention, the obtained calibration plate image and the projection image may be captured by an infrared camera, the projection module may also use a projection grating module to project a lithography pattern formed based on a binary geometric space coding pattern onto the calibration plate, and the calibrated structured light system may be a grating structured light system, so as to implement high-precision calibration of the grating structured light system with high reliability, small volume, and low cost. The method and the device for calibrating the structured light system in the embodiment of the invention can calibrate a general camera-projector structured light system, for example, the method and the device are suitable for a standard structured light system consisting of a single projector and a single camera, and can also be suitable for projection equipment with a fixed projection pattern, such as the projection grating module. The lithographic pattern projected by the projection grating module is fixed, whereas in standard structured light systems, the projector can control the variation of the projected pattern by digital means. In the calibration process of the structured light system, the projector can realize the projection pattern formed based on the binary geometric space coding pattern through a digital means. It is obvious that the structured light system of the embodiment of the present invention may be a general camera-projector structured light system, such as a standard structured light system composed of a single projector and a single camera, and may also be the aforementioned grating structured light system.
The embodiment of the invention is proved to be feasible through experiments, and the experimental object is a standard plane plate as shown in figure 6. Fig. 7 and 8 show the feature point detection result and the point cloud after three-dimensional scanning, respectively, and the distance from the point to the plane is calculated by fitting the point cloud data to the plane, thereby calculating the average error of the three-dimensional scanning to be 0.12mm and the standard error to be 0.23mm, and fig. 9 shows the curved surface reconstructed by the point cloud, and the experimental result shows that: the calibration method of the structured light system is completely feasible, and the structured light system can obtain more accurate three-dimensional information by utilizing the structured light system to perform three-dimensional scanning, which fully reflects the characteristic of high calibration precision of the structured light system in the embodiment of the invention.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (20)
1. A method for calibrating a structured light system, comprising:
obtaining a calibration plate image and a projection image of the calibration plate projected by a projection module, wherein the projection pattern of the projection module is formed based on a binary geometric space coding pattern; the binary geometric space coding pattern is generated by embedding a plurality of code word graphs into a characteristic graph by utilizing a pseudorandom array, wherein the code word graphs are black-white binary geometric graphs, the characteristic graph is a grid, lines and points in the code word graphs are white, and the background color is black;
and calibrating the structured light system comprising the camera and the projection module by utilizing the calibration plate image and the projection image.
2. The method of claim 1, wherein the projection pattern is formed by placing white in a transmissive region and black in a non-transmissive region of the binary geometric spatial coding pattern.
3. The method of claim 2, wherein the grid is formed of perpendicular intersecting lines.
4. The method of claim 2, wherein calibrating a structured light system including a camera and a projection module using the calibration plate image and the projection image comprises:
extracting coordinates of the projected image on the camera image, and establishing a corresponding relation between the camera image coordinates and the projection module image coordinates by combining the coordinates of the projected image on the projection module image;
and calibrating the structured light system by utilizing the corresponding relation and the calibration plate image.
5. The method of claim 4, wherein the extracting coordinates of the projected image on the camera image and the establishing the correspondence between the camera image coordinates and the projection module image coordinates in combination with the coordinates of the projected image on the projection module image comprises:
converting the projected image into a gray image, and smoothing the gray image by using a filter;
extracting characteristic points from the smoothed gray level image to obtain a topological structure of the characteristic points;
extracting a code word graph in the grid according to the topological structure of the feature points;
identifying the extracted code word graph and determining the code words of the characteristic points;
and determining the corresponding relation between the camera image coordinates and the projection module image coordinates according to the code word information of the identified characteristic points and the code word information of the projection characteristic points.
6. The method of claim 5, wherein identifying the extracted codeword pattern and determining codewords for the feature points comprises identifying the extracted codeword pattern using a convolutional neural network as follows:
before identification, a plurality of projection patterns containing single code word patterns are respectively projected to a calibration plate to be used for collecting training samples, the number of the samples is enlarged by adjusting Gaussian noise, executing affine transformation and applying Gaussian filtering, a sample training identification network is utilized, and the trained network is utilized to identify the extracted code word patterns.
7. The method of claim 4, wherein calibrating the structured light system using the correspondence and calibration plate images comprises:
converting the coordinates of the calibration plate on the camera image to the projection module image by utilizing the corresponding relation;
and calibrating the structured light system by using a Zhangyingyou plane calibration method.
8. The method of claim 7, wherein using the correspondence to convert coordinates of the calibration plate on the camera image to the projection module image comprises:
calculating a homography matrix between the camera image and the projection module image by using the corresponding relation;
converting the characteristic point coordinates of the calibration plate on the camera image to the projection module image by using the homography matrix;
the method for calibrating the structured light system by using the Zhangyingyou plane calibration method comprises the following steps:
and (3) calibrating the structured light system by taking the characteristic points on the calibration plate as data sources and combining a Zhangyingyou plane calibration method.
9. The method of any one of claims 1 to 8, wherein the camera is an infrared camera; the projection module is a projection grating module; the structured light system is a grating structured light system.
10. A structured light system calibration apparatus, comprising:
the image obtaining module is used for obtaining a calibration plate image and a projected image of the calibration plate projected by the projection module, wherein the projection pattern of the projection module is formed on the basis of a binary geometric space coding pattern; the binary geometric space coding pattern is generated by embedding a plurality of code word graphs into a characteristic graph by utilizing a pseudorandom array, wherein the code word graphs are black-white binary geometric graphs, the characteristic graph is a grid, and the background color is black;
and the system calibration module is used for calibrating the structured light system comprising the camera and the projection module by utilizing the calibration plate image and the projection image.
11. The apparatus of claim 10, wherein the projection pattern is formed by placing white in a transmissive mode and black in a non-transmissive mode in the binary geometric spatial coding pattern.
12. The apparatus of claim 11, wherein the grid is formed of perpendicular intersecting lines.
13. The apparatus of claim 11, wherein the system calibration module is specifically configured to:
extracting coordinates of the projected image on the camera image, and establishing a corresponding relation between the camera image coordinates and the projection module image coordinates by combining the coordinates of the projected image on the projection module image;
and calibrating the structured light system by utilizing the corresponding relation and the calibration plate image.
14. The apparatus of claim 13, wherein the system calibration module is specifically configured to:
converting the projected image into a gray image, and smoothing the gray image by using a filter;
extracting characteristic points from the smoothed gray level image to obtain a topological structure of the characteristic points;
extracting a code word graph in the grid according to the topological structure of the feature points;
identifying the extracted code word graph and determining the code words of the characteristic points;
and determining the corresponding relation between the camera image coordinates and the projection module image coordinates according to the code word information of the identified characteristic points and the code word information of the projection characteristic points.
15. The apparatus of any one of claims 10 to 14, wherein the camera is an infrared camera; the projection module is a projection grating module; the structured light system is a grating structured light system.
16. A structured light system, comprising:
the camera is used for shooting the image of the calibration plate and the projected image of the calibration plate projected by the projection module;
the projection module is used for projecting a projection pattern formed based on the binary geometric space coding pattern to the calibration plate; the binary geometric space coding pattern is generated by embedding a plurality of code word graphs into a characteristic graph by utilizing a pseudorandom array, wherein the code word graphs are black-white binary geometric graphs, the characteristic graph is a grid, and the background color is black;
the calibration plate image and the projection image are used for calibrating a structured light system comprising a camera and a projection module.
17. The structured light system of claim 16 wherein the projected pattern is formed by placing white light transparent and black light opaque in the binary geometric spatial code pattern.
18. The structured light system of claim 17 wherein the grid is formed of perpendicular intersecting lines.
19. The structured light system according to one of the claims 16 to 18, characterized in that the camera is an infrared camera;
the projection module is a projection grating module; the structured light system is a grating structured light system.
20. A mobile device, characterized by comprising the structured light system calibration apparatus of any one of claims 10 to 15 and the structured light system of any one of claims 16 to 19.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611150961.6A CN108225216B (en) | 2016-12-14 | 2016-12-14 | Structured light system calibration method and device, structured light system and mobile device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611150961.6A CN108225216B (en) | 2016-12-14 | 2016-12-14 | Structured light system calibration method and device, structured light system and mobile device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108225216A CN108225216A (en) | 2018-06-29 |
| CN108225216B true CN108225216B (en) | 2021-03-12 |
Family
ID=62637645
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611150961.6A Active CN108225216B (en) | 2016-12-14 | 2016-12-14 | Structured light system calibration method and device, structured light system and mobile device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108225216B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109141293B (en) * | 2018-08-08 | 2020-04-24 | 深圳市银星智能科技股份有限公司 | Object measuring method based on structured light and electronic equipment |
| CN110141800B (en) * | 2019-05-10 | 2020-10-16 | 苏州雷泰医疗科技有限公司 | Accelerator optical distance scale equipment, calibration method and optical distance scale generation method |
| WO2020237660A1 (en) * | 2019-05-31 | 2020-12-03 | 中国科学院深圳先进技术研究院 | Structured light-based three-dimensional reconstruction method, terminal device, and storage medium |
| CN112233183A (en) * | 2019-07-15 | 2021-01-15 | 南昌欧菲生物识别技术有限公司 | 3D structured light module support calibration method, device and equipment |
| CN110443856A (en) * | 2019-08-12 | 2019-11-12 | 广州图语信息科技有限公司 | A kind of 3D structure optical mode group scaling method, storage medium, electronic equipment |
| CN111383234B (en) * | 2020-03-04 | 2022-05-17 | 中国空气动力研究与发展中心超高速空气动力研究所 | Machine learning-based structured light online intensive three-dimensional reconstruction method |
| CN111462246B (en) * | 2020-03-09 | 2024-01-26 | 浙江未来技术研究院(嘉兴) | Equipment calibration method of structured light measurement system |
| CN111487043B (en) * | 2020-05-07 | 2021-11-09 | 北京的卢深视科技有限公司 | Method for determining calibration parameters of speckle projector of monocular speckle structured light system |
| CN111811433B (en) * | 2020-07-15 | 2022-03-08 | 河北工业大学 | Structured light system calibration method and device based on red and blue orthogonal stripes and application |
| CN112050751B (en) * | 2020-07-17 | 2022-07-22 | 深圳大学 | Projector calibration method, intelligent terminal and storage medium |
| CN112132089A (en) * | 2020-09-28 | 2020-12-25 | 天津天地伟业智能安全防范科技有限公司 | Excavator behavior analysis method based on 3D convolution and optical flow |
| CN113959362B (en) * | 2021-09-22 | 2023-09-12 | 杭州申昊科技股份有限公司 | Calibration method and inspection data processing method of structured light three-dimensional measurement system |
| CN117213401B (en) * | 2023-11-09 | 2024-02-13 | 山东捷瑞信息技术产业研究院有限公司 | Surface industry visual detection method and system based on structured light tangential scanning |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA1326774C (en) * | 1989-06-29 | 1994-02-08 | Marc Dufour | Method and apparatus for active vision image enhancement |
| CN100554873C (en) * | 2007-07-11 | 2009-10-28 | 华中科技大学 | A kind of based on two-dimensional encoded 3 D measuring method |
| CN100491903C (en) * | 2007-09-05 | 2009-05-27 | 北京航空航天大学 | Method for calibrating structural parameter of structure optical vision sensor |
| CN105987805B (en) * | 2015-02-02 | 2019-06-21 | 宁波舜宇光电信息有限公司 | A kind of detection method of projection structure photosystem |
| CN103983213B (en) * | 2014-05-30 | 2016-12-07 | 深圳先进技术研究院 | A kind of structure light coding method and relevant apparatus |
| CN104657982A (en) * | 2015-01-15 | 2015-05-27 | 华中科技大学 | Calibration method for projector |
| CN104573688B (en) * | 2015-01-19 | 2017-08-25 | 电子科技大学 | Mobile platform tobacco laser code intelligent identification Method and device based on deep learning |
-
2016
- 2016-12-14 CN CN201611150961.6A patent/CN108225216B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108225216A (en) | 2018-06-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108225216B (en) | Structured light system calibration method and device, structured light system and mobile device | |
| US10872439B2 (en) | Method and device for verification | |
| US10924729B2 (en) | Method and device for calibration | |
| US9829309B2 (en) | Depth sensing method, device and system based on symbols array plane structured light | |
| KR101974651B1 (en) | Measuring method of 3d image depth and a system for measuring 3d image depth using boundary inheritance based hierarchical orthogonal coding | |
| JPWO2018235163A1 (en) | Calibration apparatus, calibration chart, chart pattern generation apparatus, and calibration method | |
| EP3158504B1 (en) | Coded light pattern having hermitian symmetry | |
| CN106464780B (en) | XSLIT camera | |
| JP2019530059A (en) | Method for independently processing multiple target areas | |
| WO2017110293A1 (en) | Data processing apparatus and method of controlling same | |
| Tran et al. | A Structured Light RGB‐D Camera System for Accurate Depth Measurement | |
| Noury et al. | Light-field camera calibration from raw images | |
| Chen et al. | Finding optimal focusing distance and edge blur distribution for weakly calibrated 3-D vision | |
| CN116524022B (en) | Offset data calculation method, image fusion device and electronic equipment | |
| CN111160233B (en) | Human face in-vivo detection method, medium and system based on three-dimensional imaging assistance | |
| CN108895979A (en) | The structure optical depth acquisition methods of line drawing coding | |
| CN109859313B (en) | 3D point cloud data acquisition method and device, and 3D data generation method and system | |
| CN108174179B (en) | Method and computer-readable storage medium for modeling an imaging device | |
| CN115908705A (en) | Three-dimensional imaging method and device based on special codes | |
| Deglint et al. | Auto‐calibration of a projector–camera stereo system for projection mapping | |
| KR20160049639A (en) | Stereoscopic image registration method based on a partial linear method | |
| KR20190103833A (en) | Method for measuring 3-dimensional data in real-time | |
| CN114022529A (en) | Depth perception method and device based on self-adaptive binocular structured light | |
| US20200234458A1 (en) | Apparatus and method for encoding in structured depth camera system | |
| CN110853086A (en) | Depth image generation method and system based on speckle projection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |