CN118097036B - Object point cloud reconstruction system and method suitable for containing light-transmitting material - Google Patents
Object point cloud reconstruction system and method suitable for containing light-transmitting material Download PDFInfo
- Publication number
- CN118097036B CN118097036B CN202410506512.9A CN202410506512A CN118097036B CN 118097036 B CN118097036 B CN 118097036B CN 202410506512 A CN202410506512 A CN 202410506512A CN 118097036 B CN118097036 B CN 118097036B
- Authority
- CN
- China
- Prior art keywords
- points
- camera
- point cloud
- rgb
- coordinate system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 239000000463 material Substances 0.000 title claims abstract description 35
- 239000011159 matrix material Substances 0.000 claims abstract description 38
- 230000000007 visual effect Effects 0.000 claims abstract description 20
- 230000002159 abnormal effect Effects 0.000 claims abstract description 18
- 230000005540 biological transmission Effects 0.000 claims abstract description 10
- 238000013519 translation Methods 0.000 claims abstract description 10
- 238000000605 extraction Methods 0.000 claims abstract description 7
- 230000011218 segmentation Effects 0.000 claims abstract description 5
- 230000003287 optical effect Effects 0.000 claims description 16
- 238000004422 calculation algorithm Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 7
- 239000012780 transparent material Substances 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000004927 fusion Effects 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 238000001514 detection method Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 241000764238 Isis Species 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000000354 decomposition reaction Methods 0.000 claims description 2
- 239000007787 solid Substances 0.000 claims description 2
- 230000017105 transposition Effects 0.000 claims 1
- 238000004458 analytical method Methods 0.000 abstract description 2
- 238000012545 processing Methods 0.000 description 9
- 230000007547 defect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000011900 installation process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000007789 sealing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000005341 toughened glass Substances 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a system and a method for reconstructing object point cloud containing light-transmitting materials, wherein the system comprises a turntable, an RGB-D camera, a camera support, a target object and Apriltag calibration plates, the method is realized based on the system, and the system is started to automatically acquire data of different visual angles of the target object; aligning the RGB image with the depth image, extracting Apriltag corner points in the RGB image, solving a rotation matrix and a translation matrix from each corner point in a camera coordinate system to a corresponding point of the Apriltag coordinate system, and obtaining external parameters of the RGB-D camera; combining the aligned RGB images with different visual angles and depth image data, and eliminating abnormal points caused by the light-transmitting material; and (5) point cloud segmentation and target object extraction. The method can greatly reduce the error of depth information caused by strong light transmission capacity, provides an important tool for three-dimensional modeling and analysis, and is suitable for application scenes requiring high-quality point cloud data.
Description
Technical Field
The invention belongs to the technical field of three-dimensional vision, and particularly relates to an object point cloud reconstruction system and method suitable for containing light-transmitting materials.
Background
Three-dimensional vision is an important branch of computer vision, focusing on understanding and processing images and scenes in three-dimensional space, involving the use of computer and digital image processing techniques to acquire, analyze, and present information of the three-dimensional world. In recent years, due to the rapid development of three-dimensional sensing technology and explosive growth of three-dimensional geometric data, three-dimensional visual research breaks through the traditional two-dimensional image space, and analysis, understanding and interaction of the three-dimensional space are realized.
At present, an object point cloud reconstruction system containing a light transmission material mainly comprises a point cloud reconstruction system based on characteristic point matching, a voxelized reconstruction system, a three-dimensional scanner system and the like, but various defects exist in all the systems; the main defects of the point cloud reconstruction system based on characteristic point matching are as follows: 1) The calculation is complex: the feature point extraction and matching generally requires a large amount of computing resources and time, especially when processing a large-scale point cloud, which can limit the real-time performance of the system, and the area of the light-transmitting material has fewer general feature points, so that accurate matching and reconstruction are difficult to perform; 2) Noise and distortion: feature point matching is susceptible to sensor noise, illumination variation and distortion, which can lead to mismatching or instability of point cloud reconstruction; 3) Data dependency: the point cloud reconstruction performed by utilizing the characteristic point matching is characterized in that a transmission phenomenon occurs to light pulses emitted by a depth camera caused by light-transmitting materials on the surface of an object, so that a large number of abnormal points appear in the acquired point cloud data, and the abnormal points influence misjudgment of a point cloud characteristic extraction algorithm on the characteristic points, and finally, reconstruction failure or lower reconstruction precision is caused. The main drawback of voxel reconstruction is that the reconstruction accuracy is not ideal. Three-dimensional scanner systems are ideal in performance, but the main drawbacks are: the installation is complex and the price is high. In summary, there is an urgent need in the market for a system that can reconstruct three-dimensional point cloud of an object containing a light-transmitting material, which is low in cost and simple to operate.
Disclosure of Invention
In order to solve the technical problems, the invention provides an object three-dimensional reconstruction system which can greatly reduce the error of depth information of a target object caused by strong light transmission capability and has low realization cost and strong expansibility, and the object three-dimensional reconstruction system comprises a turntable, an RGB-D camera, a camera bracket, the target object and a Apriltag calibration plate.
The turntable is used for supporting and rotating the target object and presenting different angles of the target object to the camera for data acquisition. The RGB-D camera has the ability to take RGB color images and capture depth information for capturing visual data of an object. The camera mount is used to stably mount the RGB-D camera, ensuring that it remains in place and at an angle during data acquisition. The target object portion surface is composed of a light-transmitting material. Apriltag the calibration plate is placed on the turntable to provide positioning and orientation information for the image.
Based on the point cloud reconstruction system, the invention also provides an object point cloud reconstruction method suitable for containing a light-transmitting material, which comprises the following steps:
step 1, starting a point cloud reconstruction system, and automatically acquiring RGB images and depth images of different visual angles of a target object;
Step 2, aligning the RGB image with the depth image, extracting Apriltag corner points in the RGB image, solving a rotation matrix and a translation matrix from each corner point in a camera coordinate system to a corresponding point of the Apriltag coordinate system, and obtaining external parameters of the RGB-D camera;
Step 3, fusing the aligned RGB images with different visual angles and depth image data, removing noise points with Euclidean distance average value larger than a set threshold value from neighborhood points, determining the specific position of a light transmission area according to brightness gradient, and removing abnormal points caused by light transmission materials to obtain point cloud;
And 4, dividing the point cloud with the abnormal points removed by adopting a planar model dividing algorithm, and extracting a target object.
And after the point cloud reconstruction system is started in the step 1, the turntable starts to rotate at a constant speed, so that the target object placed on the turntable also rotates at a constant speed. The RGB-D camera acquires RGB images of a target object and depth images of the target object at regular intervals according to actual needs so as to acquire data of the target object at different visual angles.
And in the step 2, the alignment coefficients of the RGB image and the depth image are obtained through the physical interval and the included angle of the actual placement of the RGB sensor and the TOF sensor, so that the alignment between the RGB image and the depth image is realized, and the alignment coefficients are utilized to align the RGB image and the depth image for the data automatically acquired by each RGB-D camera. And carrying out graying treatment on the acquired RGB image, and then carrying out edge extraction to obtain the corner point of each Apriltag. The pixel coordinate system takes the upper left corner of the image as an origin, the u axis is horizontal to the right, and the v axis is vertical to the bottom. Assuming that n Apriltag corner points exist in one RGB image, the coordinates in the RGB image are as followsThe depth value corresponding to the depth image isAccording to the ID value of the identified Apriltag angular points, the three-dimensional coordinate value of the angular points in a Apriltag coordinate system is obtainedThe z coordinate is 0, the origin of the apriltag coordinate system is positioned at the left lower corner of the Apriltag calibration plate, the x direction and the y direction are respectively overlapped with the two sides of the initial position of the Apriltag calibration plate, the z axis is vertical to the plane of the calibration plate and upwards, and the coordinate system accords with the right-hand coordinate system.
The coordinate origin of the camera coordinate system is positioned at the optical center of the camera, the X axis points to the right side of the camera and is perpendicular to the optical axis and parallel to the image plane, the Y axis points to the upper part of the camera and is perpendicular to the optical axis and parallel to the X axis and the image plane, and the Z axis points to the observation direction of the camera, namely points to the observed scene and is parallel to the optical axis; and (3) obtaining a corner depth value by using a depth map aligned with the RGB image, and converting a corner coordinate from a pixel coordinate system to a camera coordinate system by combining with an internal camera parameter, wherein a coordinate conversion formula is as follows:
（1）
Wherein u and v are coordinate values of the pixel point in a pixel coordinate system, Respectively represent the focal lengths of the cameras in the u and v directions in the pixel coordinate system,The pixel coordinates of the principal point are X, Y, Z, which are coordinate values of the pixel point in the camera coordinate system, and the Z value of the pixel point in the camera coordinate system is the depth value d.
The external parameters of the camera refer to the pose of the cameraIncluding a rotation matrix R and a translation matrix t, assuming that there are multiple rays, each of which connects the camera optical center, the three-dimensional target point, and the projection of the target point onto the camera plane, for points in the camera coordinate systemRepresentation, for points in Apriltag coordinate systemThe camera external parameters are solved, namely a rotation matrix and a translation matrix of each point in the camera coordinate system to a corresponding point of the Apriltag coordinate system are solved; external parameter adjustment of cameraSuch that the expressionThe value of (2) is obtained by the minimum value,Representing the square of the 2-norm.
In the step 3, the aligned RGB image and depth image are firstly converted into Apriltag coordinate system by using camera external parameters, and then the RGB image and depth image data acquired from the viewing angle are converted into Apriltag coordinate system at the initial time according to the angle of Apriltag at the current time compared with the angle of rotation at the initial time, so as to realize multi-angle view fusion. Calculating Euclidean distance between each point in the point cloud data obtained by fusing the RGB images and the depth images with different visual angles and the point in the K neighborhood of the point cloud data, and calculating the average value of all the Euclidean distancesAnd standard deviationTaking a distance thresholdWhereinTraversing the point cloud again with the constant, namely the proportionality coefficient, and removing that the average value of Euclidean distances between the point cloud and K neighborhood points is larger thanIs a point of (2). For abnormal points caused by transparent materials, firstly extracting characteristic points in an RGB image by using a Harris characteristic point detection method, uniformly dividing the characteristic points into N subsets, then calculating the intensity change direction of the surrounding area of each characteristic point in the subsets, taking each characteristic point as an origin, making rays with the direction pointing to the brightness becoming larger, if a certain number of rays exist in the subsets and intersect at a common point, recognizing a closed area formed by connecting the points as a transparent material area, deleting the characteristic points in the area, judging and deleting the characteristic points in the subsets one by one, and removing abnormal data.
And in the step 4, the RGB images and depth information of each angle of the target object continuously collected in the step 1-3 are unified under a Apriltag coordinate system at the initial moment, and after the data processing of abnormal points caused by the transparent material is removed, the multi-angle fused point cloud data are obtained. Screening each point cloud, removing points with z coordinate values smaller than 0 and x and y larger than the size of the calibration plate in Aptiltag coordinate system to obtain point cloud information of a target object and a turntable, adopting a planar model segmentation algorithm to segment the fused point cloud data into voxels under the condition that the turntable with a contact area with the target object is difficult to segment, only reserving central points of each voxel, setting iteration times threshold, distance threshold and inner point number threshold, randomly selecting central points of M voxels, fitting a plane equation by utilizing Ransac algorithm, and estimating parameters of a plane model. Calculating the distance from the central point of all voxels to the estimated plane, taking the point with the distance smaller than the threshold value as the inner point, counting the number of the inner points, reaching the appointed iteration times or the number of the inner points reaching the set threshold value, and stopping the iteration. And removing the inner points in the plane model, namely removing the turntable point cloud, and finally obtaining the three-dimensional target object model.
Compared with the prior art, the invention has the following advantages:
1) Through automatic rotation of the turntable and periodic image acquisition, automatic data acquisition is realized, operation intervention of a user is reduced, data acquisition is more efficient, and the real-time performance of a point cloud reconstruction system and the consistency of data acquisition are improved;
2) By analyzing Apriltag tag information in the captured image, external parameters of the camera, including position and direction information, are calculated, a reliable basis is provided for subsequent data processing and point cloud splicing, and the accuracy of point cloud reconstruction is improved;
3) The images of different angles are obtained by rotating the target object, and multi-angle view fusion is carried out to obtain a more comprehensive object view, so that shielding is reduced, more geometric information is provided, and the accuracy and the integrity of point cloud reconstruction are improved;
4) The interference of the light-transmitting material of the target object on the depth information is considered, and the accuracy of the point cloud data is ensured by detecting and removing points of abnormal depth values caused by strong light-transmitting capacity, so that the problems of noise and distortion of characteristic point matching are solved;
5) Compared with other point cloud reconstruction systems in the market, the point cloud reconstruction system provided by the invention has the advantages that the structure is relatively simple, the used equipment is relatively common, and the complex installation process is not needed, so that the cost of the system is reduced, and the convenience in operation is provided.
In summary, the invention solves various problems existing in the current point cloud reconstruction system through technical innovation, and provides the object point cloud reconstruction system and method which are efficient, accurate, low in cost and simple to operate and are suitable for the object point cloud reconstruction system containing the light-transmitting material.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a block diagram of a point cloud reconstruction system for an object containing a light transmissive material according to an embodiment of the present invention.
Fig. 2 is a flowchart of a method for reconstructing an object point cloud containing a light-transmitting material according to an embodiment of the present invention.
Fig. 3 is a flowchart of a camera exogenous estimation method according to an embodiment of the present invention.
FIG. 4 is a flow chart of a light transmissive material data processing in an embodiment of the invention.
Fig. 5 is a flowchart of a point cloud segmentation algorithm according to an embodiment of the present invention.
Detailed Description
In order that the application may be readily understood, a more complete description of the application will be rendered by reference to the appended drawings. Preferred embodiments of the present application are shown in the drawings. This application may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Furthermore, the examples are provided for a more thorough and complete disclosure of the present application.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
Example 1
As shown in fig. 1, embodiment 1 of the present invention provides an object point cloud reconstruction system suitable for use with light transmissive materials, comprising a turntable, an RGB-D camera, a camera stand, an automobile model, and Apriltag calibration plates.
The turntable is used for supporting and rotating the target object and presenting different angles of the target object to the camera for data acquisition. RGB-D cameras have RGB (color image) and depth (distance information) capturing capabilities for capturing visual data of an object. The camera mount is used to stably mount the RGB-D camera, ensuring that it remains in place and at an angle during data acquisition. The surface of a part of the target object is made of a light-transmitting material, and the target object in the embodiment is an automobile model and comprises a toughened glass window with light-transmitting property. Apriltag the calibration plate is placed on the turntable to provide positioning and orientation information for the image. The whole point cloud reconstruction system needs to be placed in front of a solid background.
Example 2
Based on the object point cloud reconstruction system provided in embodiment 1, embodiment 2 of the present invention further provides a method for reconstructing an object point cloud containing a light-transmitting material, as shown in fig. 2, including the following steps:
Step 1, automatically acquiring data of different visual angles of a target object.
After the object point cloud reconstruction system is started, the turntable starts to rotate at a constant speed, so that the automobile model placed on the turntable also rotates at a constant speed. The RGB-D camera acquires an RGB image of an automobile model and a depth image of the automobile model at regular intervals according to actual needs to acquire data of the automobile model at different visual angles for subsequent processing.
And 2, aligning the RGB image with the depth image, extracting Apriltag corner points in the RGB image, solving a rotation matrix and a translation matrix from each corner point in a camera coordinate system to a corresponding point in the Apriltag coordinate system, and obtaining external parameters of the RGB-D camera.
As shown in fig. 3, embodiment 2 of the present invention further provides a camera external parameter estimation method, which specifically includes the following steps:
Since there is a certain physical distance between the RGB sensor and the TOF sensor in the RGB-D camera, the alignment coefficients of the RGB image and the depth image need to be obtained through the physical interval and the included angle between the actual placement of the RGB sensor and the TOF sensor, so as to realize the alignment between the two. For each data auto-acquired by the RGB-D camera, the RGB image and the depth image need to be aligned using an alignment coefficient.
And carrying out graying treatment on the acquired RGB image, and then carrying out edge extraction to obtain the corner point of each Apriltag. The pixel coordinate system takes the upper left corner of the image as an origin, the u axis is horizontal to the right, and the v axis is vertical to the bottom. Assuming 15 complete Apriltag appear in an RGB image, each Apriltag containing 4 corners, there are 60 Apriltag corners, and the pixel coordinates in the RGB image areThe depth value corresponding to the depth image is. According to the ID value of the identified Apriltag angular points, the three-dimensional coordinate value of the angular points in a Apriltag coordinate system is obtainedThe z coordinate is 0. The origin of Apriltag coordinate system is located at the left lower corner of Apriltag calibration plate, x direction and y direction coincide with the both sides length of Apriltag calibration plate initial position respectively, and the z axis is perpendicular to calibration plate plane face up, and this coordinate system accords with right hand coordinate system.
The origin of coordinates of the camera coordinate system is located at the optical center of the camera (the intersection point of the optical axes), the X-axis points to the right side of the camera, is perpendicular to the optical axis and parallel to the image plane, the Y-axis points to the upper side of the camera, is perpendicular to the optical axis and parallel to the X-axis and the image plane, and the Z-axis points to the observation direction of the camera, i.e., points to the observed scene and is parallel to the optical axis; and (3) obtaining a corner depth value by using a depth map aligned with the RGB image, and converting a corner coordinate from a pixel coordinate system to a camera coordinate system by combining with an internal camera parameter, wherein a coordinate conversion formula is as follows:
（1）
Wherein u and v are coordinate values of the pixel point in a pixel coordinate system, Respectively represent the focal lengths of the cameras in the u and v directions in the pixel coordinate system,The pixel coordinates of the principal point are X, Y, Z, which are coordinate values of the pixel point in the camera coordinate system, and the Z value of the pixel point in the camera coordinate system is the depth value d.
The external parameters of the camera refer to the pose of the cameraIncluding a rotation matrix R and a translation matrix t. Assuming that there are multiple rays, each of which connects the camera optical center, the three-dimensional target point, and the projection of the target point onto the camera plane, the points in the camera coordinate system are usedRepresentation, for points in Apriltag coordinate systemThe camera external parameters are solved, namely a rotation matrix and a translation matrix of each point in the camera coordinate system to a corresponding point of the Apriltag coordinate system are solved; external parameter adjustment of cameraSuch that the expressionThe value of (2) is obtained by the minimum value,Representing the square of the 2-norm. The present embodiment uses Bundle Adjustment (BA) nonlinear optimization techniques to solve the outliers. BA improves the accuracy of the estimation by minimizing the re-projection error, enabling high-precision conversion.
And 3, fusing the aligned RGB images with different visual angles and depth image data, removing noise points with the Euclidean distance average value of the adjacent points larger than a set threshold value, determining the specific position of the light transmission area according to the brightness gradient, and removing abnormal points caused by the light transmission material.
As shown in fig. 4, embodiment 2 of the present invention further provides a method for processing data of a light-transmitting material, which specifically includes:
Firstly, the aligned RGB image and depth image are converted into Apriltag coordinate system by utilizing camera external parameters, then according to the angle rotated by the current time Apriltag compared with the initial time, the RGB image and depth image data acquired by the visual angle are converted into Apriltag coordinate system at the initial time, so as to realize multi-angle view fusion. Because the light transmittance window of the automobile model may interfere the depth information collected by the TOF camera, based on the continuous sealing characteristic of the automobile model, the Euclidean distance (20 is taken by K in the embodiment, and can be taken according to the situation in the implementation) between each point in the point cloud data obtained by fusing the RGB images and the depth images with each other in multiple different visual angles and the point cloud data in the K neighborhood of the point cloud data is calculated, and the average value of all the Euclidean distances is calculated And standard deviationTaking a distance thresholdWhereinTraversing the point cloud again with a constant, i.e. proportional coefficient, removing the average value of Euclidean distances between the point cloud and 20 neighborhood points being greater thanIs a point of (2).
For abnormal points caused by transparent materials, firstly, extracting characteristic points in an RGB image by using a Harris characteristic point detection method, uniformly dividing the characteristic points into N subsets (N is 10 according to the situation in the embodiment), then calculating the intensity change direction of the area around each characteristic point in the subset, taking each characteristic point as an origin, directing a ray with increased brightness in the direction, and if a certain number of rays exist in the subset and intersect at a common point, identifying a closed area formed by connecting the points as a transparent material area, and deleting the characteristic points in the area. And carrying out the judging and deleting operation on the characteristic points in the subsets one by one, and eliminating abnormal data.
And 4, point cloud segmentation and target object extraction.
And (3) continuously collecting RGB images and depth information of each angle of the target object through the steps 1-3, unifying the RGB images and the depth information to a Apriltag coordinate system at the initial moment, and removing data processing of abnormal points caused by the transparent material to obtain multi-angle fused point cloud data.
In order to extract the automobile model from the scene, screening each point cloud, and removing points with z coordinate values smaller than 0 and x and y larger than the size of the calibration plate in a Aptiltag coordinate system, so as to obtain the point cloud information of the target object and the turntable. For the situation that the turntable with the contact area with the automobile model is difficult to divide, a plane model dividing algorithm is adopted, specific operations are shown in fig. 5, the fused point cloud data are divided into voxels, only the center point of each voxel is reserved, an iteration number threshold, a distance threshold and an inner point number threshold (the maximum iteration number is 1000 in the embodiment, the distance threshold is 0.01, the inner point number threshold is 30% of the original point cloud number) are set, the center points of M voxels are randomly selected (M in the embodiment is 50 according to the situation in the specific implementation), the plane equation is fitted by using Ransac algorithm, and parameters of the plane model are estimated. Assume thatIs a set of M voxel center points,Is any point in the set of points P,Is a set of pointsCenter of (i.e.)Assuming that the fitted object plane passesAnd the normal vector isSolving by least square method to obtainMake the following stepsThe distance of all points from the plane, i.eThe value of (2) is the smallest. Definition matrixThen:
（2）
in the method, in the process of the invention, Representing the norm of the matrix and the superscript T representing the transpose of the matrix.
Singular value decomposition is carried out on the matrix A to obtainSubstituting formula (2) to obtain:
（3）
wherein V is Is a unitary matrix of (a); u isIs a unitary matrix of (a); Is one Each element on the main diagonal is a singular value, all 0 except the element on the main diagonal, i.e。
Will beUsing oneMatrix W represents, i.eThen formula (3) is expressed as:
（4）
in the method, in the process of the invention, 、、Are all singular values, and；、、Is an element of the matrix W.
Finally, a fitted plane normal vector is obtainedFor the third column of the matrix U, i.e。
Calculating the distance from the central point of all voxels to the estimated plane, taking the point with the distance smaller than the threshold value as the inner point, counting the number of the inner points, reaching the appointed iteration times or the number of the inner points reaching the set threshold value, and stopping the iteration. And removing the inner points in the plane model, namely removing the turntable point cloud, and finally obtaining the three-dimensional target object model.
In particular, the method according to the technical solution of the present invention may be implemented by those skilled in the art using computer software technology to implement an automatic operation flow, and a system apparatus for implementing the method, such as a computer readable storage medium storing a corresponding computer program according to the technical solution of the present invention, and a computer device including the operation of the corresponding computer program, should also fall within the protection scope of the present invention.
The specific embodiments described herein are offered by way of example only to illustrate the spirit of the invention. Those skilled in the art may make various modifications or additions to the described embodiments or substitutions thereof without departing from the spirit of the invention or exceeding the scope of the invention as defined in the accompanying claims.
Claims (8)
1. A method for reconstructing object point cloud containing light-transmitting material is characterized by comprising the steps of,
The object point cloud reconstruction process is realized based on a point cloud reconstruction system, wherein the point cloud reconstruction system comprises a turntable, an RGB-D camera, a camera bracket, a target object and Apriltag calibration plates;
The turntable is used for supporting and rotating the target object, and presenting different angles of the target object to the camera for data acquisition; the RGB-D camera has the capabilities of shooting RGB color images and capturing depth information and is used for acquiring visual data of an object; the camera support is used for stably mounting the RGB-D camera; the surface of the target object part is made of a light-transmitting material; apriltag the calibration plate is placed on the turntable to provide positioning and orientation information for the image; the whole point cloud reconstruction system needs to be placed in front of a solid background;
The object point cloud reconstruction process comprises the following steps:
step 1, starting a point cloud reconstruction system, and automatically acquiring RGB images and depth images of different visual angles of a target object;
Step 2, aligning the RGB image with the depth image, extracting Apriltag corner points in the RGB image, solving a rotation matrix and a translation matrix from each corner point in a camera coordinate system to a corresponding point of the Apriltag coordinate system, and obtaining external parameters of the RGB-D camera;
Step 3, fusing the aligned RGB images with different visual angles and depth image data, removing noise points with Euclidean distance average value larger than a set threshold value from neighborhood points, determining the specific position of a light transmission area according to brightness gradient, and removing abnormal points caused by light transmission materials to obtain point cloud;
firstly, converting an aligned RGB image and depth image into a Apriltag coordinate system by utilizing camera external parameters, and then converting RGB image and depth image data acquired by the visual angle into a Apriltag coordinate system at the initial moment according to the angle which is compared with Apriltag at the current moment and is turned over at the initial moment, so as to realize multi-angle view fusion; calculating Euclidean distance between each point in the point cloud data obtained by fusing the RGB images and the depth images with different visual angles and the point in the K neighborhood of the point cloud data, and calculating the average value of all the Euclidean distances And standard deviationTaking a distance thresholdWhereinTraversing the point cloud again with the constant, namely the proportionality coefficient, and removing that the average value of Euclidean distances between the point cloud and K neighborhood points is larger thanIs a point of (2); for abnormal points caused by light-transmitting materials, firstly extracting characteristic points in an RGB image by using a Harris characteristic point detection method, uniformly dividing the characteristic points into N subsets, then calculating the intensity change direction of the surrounding area of each characteristic point in the subsets, taking each characteristic point as an origin, making rays with direction pointing to the brightness becoming larger, if a certain number of rays exist in the subsets and intersect at a common point, recognizing a closed area formed by connecting the points as a light-transmitting material area, deleting the characteristic points in the area, judging and deleting the characteristic points in the subsets one by one, and eliminating abnormal data;
And 4, dividing the point cloud with the abnormal points removed by adopting a planar model dividing algorithm, and extracting a target object.
2. A method for reconstructing an object point cloud comprising a light transmissive material as recited in claim 1, wherein: after the point cloud reconstruction system is started in the step 1, the turntable starts to rotate at a constant speed, so that a target object placed on the turntable also rotates at a constant speed; the RGB-D camera acquires RGB images of a target object and depth images of the target object at regular intervals according to actual needs so as to acquire data of the target object at different visual angles.
3. A method for reconstructing an object point cloud comprising a light transmissive material as recited in claim 1, wherein: in the step 2, the alignment coefficients of the RGB image and the depth image are obtained through the physical interval and the included angle of the actual arrangement of the RGB sensor and the TOF sensor, so that the alignment between the RGB image and the depth image is realized, and the alignment coefficients are utilized to align the RGB image and the depth image for the data automatically acquired by each RGB-D camera; carrying out graying treatment on the acquired RGB image, and then carrying out edge extraction to obtain each Apriltag corner points; the pixel coordinate system takes the upper left corner of the image as the origin, the u axis is horizontal to the right, the v axis is vertical to the bottom, n Apriltag corner points are assumed to exist in one RGB image, and the pixel coordinate in the RGB image isThe depth value corresponding to the depth image isAccording to the ID value of the identified Apriltag angular points, the three-dimensional coordinate value of the angular points in a Apriltag coordinate system is obtainedThe z coordinate is 0, the origin of the apriltag coordinate system is positioned at the left lower corner of the Apriltag calibration plate, the x direction and the y direction are respectively overlapped with the two sides of the initial position of the Apriltag calibration plate, the z axis is vertical to the plane of the calibration plate and upwards, and the coordinate system accords with the right-hand coordinate system.
4. A method for reconstructing an object point cloud comprising a light transmissive material as recited in claim 3, wherein: in the step 2, the origin of coordinates of the camera coordinate system is located at the optical center of the camera, the X axis points to the right side of the camera and is perpendicular to the optical axis, parallel to the image plane, the Y axis points to the upper side of the camera and is perpendicular to the optical axis, parallel to the X axis and the image plane, and the Z axis points to the observation direction of the camera, namely points to the observed scene, and is parallel to the optical axis; and (3) obtaining a corner depth value by using a depth map aligned with the RGB image, and converting a corner coordinate from a pixel coordinate system to a camera coordinate system by combining with an internal camera parameter, wherein a coordinate conversion formula is as follows:
（1）
Wherein u and v are coordinate values of the pixel point in a pixel coordinate system, Respectively represent the focal lengths of the cameras in the u and v directions in the pixel coordinate system,The pixel coordinate of the main point is X, Y, Z, which is the coordinate value of the pixel point in the camera coordinate system, and the Z value of the pixel point in the camera coordinate system is the depth value d;
The external parameters of the camera refer to the pose of the camera Including a rotation matrix R and a translation matrix t, assuming that there are multiple rays, each of which connects the camera optical center, the three-dimensional target point, and the projection of the target point onto the camera plane, for points in the camera coordinate systemRepresentation, for points in Apriltag coordinate systemThe camera external parameters are solved, namely a rotation matrix and a translation matrix of each point in the camera coordinate system to a corresponding point of the Apriltag coordinate system are solved; external parameter adjustment of cameraSuch that the expressionThe value of (2) is obtained by the minimum value,Representing the square of the 2-norm.
5. A method for reconstructing an object point cloud comprising a light transmissive material as recited in claim 1, wherein: in the step 4, the RGB images and depth information of all angles of the target object which are continuously collected in the step 1 are unified to a Apriltag coordinate system at the initial moment, and after the data of abnormal points caused by the transparent material are removed, the multi-angle fused point cloud data are obtained; screening each point cloud, removing points with z coordinate values smaller than 0 and x and y larger than the size of the calibration plate in Aptiltag coordinate system to obtain point cloud information of a target object and a turntable, adopting a plane model segmentation algorithm to segment the fused point cloud data into voxels under the condition that the turntable with a contact area with the target object is difficult to segment, only reserving central points of each voxel, setting iteration times threshold, distance threshold and inner point number threshold, randomly selecting central points of M voxels, fitting a plane equation by utilizing Ransac algorithm, and estimating parameters of a plane model; calculating the distances from the central points of all voxels to the estimated plane, taking the points with the distances smaller than the threshold value as internal points, counting the number of the internal points, and stopping iteration when the number of the specified iteration times or the number of the internal points reaches the set threshold value; and removing the inner points in the plane model, namely removing the turntable point cloud, and finally obtaining the three-dimensional target object model.
6. A method for reconstructing an object point cloud comprising a light transmissive material as recited in claim 5, wherein: in step 4, the calculation process of estimating the parameters of the plane model by using Ransac algorithm fitting plane equation is as follows:
Assume that Is a set of M voxel center points,Is any point in the set of points P,Is a set of pointsCenter of (i.e.)Assuming that the fitted object plane passesAnd the normal vector isSolving by least square method to obtainMake the following stepsThe distance of all points from the plane, i.eMinimum value of (2) defining a matrixThen:
（2）
in the method, in the process of the invention, Representing norms of the matrix, and superscript T represents matrix transposition;
Singular value decomposition is carried out on the matrix A to obtain Substituting formula (2) to obtain:
（3）
wherein V is Is a unitary matrix of (a); u isIs a unitary matrix of (a); Is one Each element on the main diagonal is a singular value, all 0 except the element on the main diagonal, i.e；
Will beUsing oneMatrix W represents, i.eThen formula (3) is expressed as:
（4）
in the method, in the process of the invention, 、、Are all singular values, and；、、Is an element of matrix W;
The resulting fitted planar normal vector And is the third column of the matrix U.
7. An object point cloud reconstruction device adapted for use with a light transmissive material, comprising a processor and a memory for storing program instructions, the processor being adapted to invoke the program instructions in the memory to perform a method as claimed in any of claims 1-6 adapted for use with an object point cloud reconstruction device comprising a light transmissive material.
8. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when executed, implements a method for reconstructing an object point cloud comprising light transmitting material according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410506512.9A CN118097036B (en) | 2024-04-25 | 2024-04-25 | Object point cloud reconstruction system and method suitable for containing light-transmitting material |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410506512.9A CN118097036B (en) | 2024-04-25 | 2024-04-25 | Object point cloud reconstruction system and method suitable for containing light-transmitting material |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118097036A CN118097036A (en) | 2024-05-28 |
| CN118097036B true CN118097036B (en) | 2024-07-12 |
Family
ID=91157675
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410506512.9A Active CN118097036B (en) | 2024-04-25 | 2024-04-25 | Object point cloud reconstruction system and method suitable for containing light-transmitting material |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118097036B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106952341B (en) * | 2017-03-27 | 2020-03-31 | 中国人民解放军国防科学技术大学 | Underwater scene three-dimensional point cloud reconstruction method and system based on vision |
| AU2021230443A1 (en) * | 2020-03-06 | 2022-04-28 | Yembo, Inc. | Identifying flood damage to an indoor environment using a virtual representation |
| CN114782628A (en) * | 2022-04-25 | 2022-07-22 | 西安理工大学 | Indoor real-time three-dimensional reconstruction method based on depth camera |
| CN115880443B (en) * | 2023-02-28 | 2023-06-06 | 武汉大学 | Implicit surface reconstruction method and implicit surface reconstruction equipment for transparent object |
| CN116758223A (en) * | 2023-07-06 | 2023-09-15 | 哈尔滨工业大学 | Three-dimensional multispectral point cloud reconstruction method, system and equipment based on double-angle multispectral image |
-
2024
- 2024-04-25 CN CN202410506512.9A patent/CN118097036B/en active Active
Non-Patent Citations (2)
| Title |
|---|
| "Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects";Haoping Xu等;《5th Conference on Robot Learning (CoRL 2021), London, UK.》;20210930;4 * |
| "双目视差图到点云的转换（原理+代码）";Edvincecilia;《https://blog.csdn.net/qq_41037856/article/details/134701111》;20240222;1-3 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118097036A (en) | 2024-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109242913B (en) | Method, device, equipment and medium for calibrating relative parameters of collector | |
| CN107705333B (en) | Space positioning method and device based on binocular camera | |
| CN111339951A (en) | Body temperature measuring method, device and system | |
| CN109685078B (en) | Infrared image identification method based on automatic annotation | |
| CN113689578B (en) | Human body data set generation method and device | |
| CN109472829B (en) | Object positioning method, device, equipment and storage medium | |
| CN112396640A (en) | Image registration method and device, electronic equipment and storage medium | |
| CN110400315A (en) | A kind of defect inspection method, apparatus and system | |
| US20200258300A1 (en) | Method and apparatus for generating a 3d reconstruction of an object | |
| CN113516702B (en) | Method and system for detecting liquid level of automatic liquid preparation ampoule bottle and method for detecting proportion of liquid medicine | |
| WO2020207172A1 (en) | Method and system for optical monitoring of unmanned aerial vehicles based on three-dimensional light field technology | |
| CN116704048B (en) | Double-light registration method | |
| CN115222884A (en) | Space object analysis and modeling optimization method based on artificial intelligence | |
| CN109214350B (en) | Method, device and equipment for determining illumination parameters and storage medium | |
| CN111899345B (en) | Three-dimensional reconstruction method based on 2D visual image | |
| CN114485953A (en) | Temperature measuring method, device and system | |
| WO2023065721A1 (en) | Methods, devices and systems for transparent object three-dimensional reconstruction | |
| US20220405968A1 (en) | Method, apparatus and system for image processing | |
| Zhao et al. | 3DRef: 3D Dataset and Benchmark for Reflection Detection in RGB and Lidar Data | |
| CN118097036B (en) | Object point cloud reconstruction system and method suitable for containing light-transmitting material | |
| JPH04130587A (en) | Three-dimensional picture evaluation device | |
| Farhood et al. | 3D point cloud reconstruction from a single 4D light field image | |
| CN115131459B (en) | Reconstruction method and device for floor plan | |
| CN116343155A (en) | Determination method and device for travelable area under BEV visual angle | |
| CN116205777A (en) | Control method and control device for glass installation and high-altitude mechanical equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |