CN115100350A - Monocular vision-based oral modeling method, system and storage medium - Google Patents
Monocular vision-based oral modeling method, system and storage medium Download PDFInfo
- Publication number
- CN115100350A CN115100350A CN202210676629.2A CN202210676629A CN115100350A CN 115100350 A CN115100350 A CN 115100350A CN 202210676629 A CN202210676629 A CN 202210676629A CN 115100350 A CN115100350 A CN 115100350A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- point
- cloud data
- calibration object
- points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Graphics (AREA)
- Biophysics (AREA)
- Geometry (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Dental Tools And Instruments Or Auxiliary Dental Instruments (AREA)
Abstract
The invention discloses an oral cavity modeling method, an oral cavity modeling system and a storage medium based on monocular vision, wherein a calibration object is attached to the front surface of gum, then an oral cavity image with the calibration object is obtained by adopting a monocular camera to generate first point cloud data containing characteristics of the oral cavity and the calibration object, then outliers and interference information of the first point cloud data are removed, the first point cloud data are separated to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data, the calibration object point cloud data are compared with a pre-stored actual calibration object model to determine a scaling factor of the calibration object, and a tooth three-dimensional model is obtained by reconstructing the tooth point cloud data according to the scaling factor. The whole intraoral scanning process can be completed only by depending on a single-sided camera, and the problems that the cost of the conventional intraoral scanner for three-dimensional reconstruction of teeth in an oral cavity by depending on binocular reconstruction or RGBD (red green blue direct) mode is high and miniaturization is difficult are solved.
Description
Technical Field
The invention relates to the technical field of intelligent medical treatment, in particular to a monocular vision-based oral cavity modeling method, a monocular vision-based oral cavity modeling system and a storage medium.
Background
In the clinical oral diagnosis and treatment and repair process, the method for obtaining the three-dimensional digital model of the interior of the oral cavity through three-dimensional scanning mainly comprises two types of extraoral scanning and intraoral scanning. The extraoral scanning mode requires that a doctor firstly takes an impression of an oral cavity to obtain a plaster model of a tooth, and then utilizes three-dimensional scanning equipment to scan the plaster model of the tooth to obtain a three-dimensional digital model of the tooth; the intraoral scanning mode is to extend the oral scanner into the oral cavity to directly scan the teeth so as to obtain the three-dimensional digital model of the teeth, and the intraoral scanning mode has the advantages of simple operation, high efficiency and high measurement speed, thereby saving the time of chair-side operation of doctors, avoiding errors caused by mold making and mold overturning due to no need of manual impression, and having higher measurement precision. At present, most intraoral scanning technologies perform three-dimensional reconstruction on teeth in an oral cavity through a binocular reconstruction or RGBD (red, green and blue) mode to obtain a digital oral cavity three-dimensional model. The binocular camera and the RGBD sensor can measure distances through technologies such as a disparity map and structured light respectively to obtain scales. However, such oral cavity scanning devices using binocular cameras or RGBD sensors for three-dimensional reconstruction are expensive and difficult to miniaturize. And if the monocular three-dimensional reconstruction can be carried out through the monocular camera, the cost and the volume of the scanner can be effectively reduced, but due to the limitation of the monocular three-dimensional reconstruction, the actual scale of the model cannot be determined by pictures.
Disclosure of Invention
The invention aims to solve the problems and discloses a monocular vision-based oral modeling method, a monocular vision-based oral modeling system and a storage medium, wherein the method comprises the following steps:
s1, acquiring an oral cavity image with a calibration object through the monocular camera, wherein the calibration object is attached to the front surface of the gum, and first point cloud data containing characteristics of the oral cavity and the calibration object are generated;
s2, classifying the first point cloud data through a first neural network classifier to form a first point type and a second point type, wherein the first point type is a point cloud attached to the surface of a tooth, a gum or a calibration object, and the second point type is a discrete point cloud;
s3, traversing each frame of oral cavity image, deleting the feature points and constraint relations in each frame of image as second-class points, taking the feature points in all the remaining frame images as candidate feature points of the first-class points, deleting the whole frame of image which does not contain the first-class points, and acquiring second point cloud data consisting of the first-class points;
s4, screening and classifying the second point cloud data through a second neural network classifier, and respectively obtaining tooth point cloud data, calibration object point cloud data and gum point cloud data;
and S5, comparing the point cloud data of the calibration object with a pre-stored actual model of the calibration object to determine a scaling factor of the calibration object, and reconstructing the point cloud data of the teeth according to the scaling factor to obtain a three-dimensional model of the teeth.
Preferably, the step S2 includes: the method comprises the steps of obtaining and marking first point cloud data serving as a sample, marking point clouds attached to teeth, gum or the surface of a calibration object as first-class points, marking discrete point clouds as second-class points, and sending the marked first-class points and second-class points into a first neural network classifier for training to screen the first-class points.
Preferably, the step S2 further includes:
judging each feature point in the first point cloud data serving as the sample respectively, if the feature point is larger than the surrounding field, reserving the feature point, and otherwise, taking the adjacent feature point on the right side as a maximum candidate point;
searching rightwards by adopting a monotone increasing mode until a characteristic point meeting I [ I ] > I [ I +1] is found, and if I < ═ W-1, reserving the point;
calculating the characteristic point quality
If the point quality is greater than a certain threshold value, retaining the point and multiplying by the contribution weight of the surrounding points to the point, otherwise setting the point qualityIs zero;
and judging whether the feature point has redundancy with the feature point at the position adjacent to the previous frame and the next frame, if so, deleting the feature point, and otherwise, keeping and marking the feature point as a first type point.
Preferably, the step S4 includes: the calibration object is a cuboid calibration object, second point cloud data serving as a sample are obtained, different labels are respectively carried out on point clouds belonging to the cuboid calibration object and teeth in the second point cloud data, and the second point cloud data after being labeled are sent to a second neural network classifier to be trained and used for screening the point cloud data of the calibration object and the point cloud data of the teeth.
Preferably, the step S5 includes:
carrying out uniform sampling on a three-dimensional model of a calibration object independently to obtain first sampling data;
performing plane extraction on the first sampling data, and acquiring point data of four planes of the calibration object which are contacted with each other in pairs as point cloud data of adjacent surfaces;
adding a plane regularization constraint to the first sampling data to obtain second sampling data, wherein the plane regularization constraint is that the distance from each surface association point of the adjacent surface point cloud data to the current surface is minimum, the included angle of two surfaces containing less point cloud data approaches to zero, and the included angle of two surfaces containing less point cloud data and two surfaces containing more point cloud data approaches to 90 degrees;
and comparing the second sampling data with the point cloud data of the calibration object to determine a scaling factor of the calibration object.
The invention also discloses an oral cavity scanning system based on monocular vision, which comprises a single-face camera and a controller connected with the monocular camera, wherein the controller is configured to: acquiring an oral cavity image with a calibration object through a monocular camera, wherein the calibration object is attached to the front surface of the gum, and first point cloud data containing characteristics of the oral cavity and the calibration object are generated; classifying the first point cloud data through a first neural network classifier to form a first point type and a second point type, wherein the first point type is a point cloud attached to the surface of a tooth, a gum or a calibration object, and the second point type is a discrete point cloud; traversing each frame of oral cavity image, deleting the characteristic points and constraint relations in each frame of image as second-class points, taking the characteristic points in all the remaining frame of image as candidate characteristic points of the first-class points, deleting the whole frame of image which does not contain the first-class points, and acquiring second point cloud data consisting of the first-class points; screening and classifying the second point cloud data through a second neural network classifier, and respectively acquiring tooth point cloud data, calibration object point cloud data and gum point cloud data; and comparing the point cloud data of the calibration object with a pre-stored actual model of the calibration object to determine a scaling factor of the calibration object, and reconstructing the point cloud data of the teeth according to the scaling factor to obtain a three-dimensional model of the teeth.
Preferably, the controller is further configured to label the acquired first point cloud data as a sample, label the point cloud attached to the tooth, gum or surface of the calibration object as a first type of point, label the discrete point cloud as a second type of point, and send the labeled first type of point and second type of point to the first neural network classifier for training to screen the first type of point.
Preferably, the controller is further configured to obtain second point cloud data serving as a sample, perform different labeling on point clouds belonging to a cuboid calibration object and teeth in the second point cloud data, and send the labeled second point cloud data to a second neural network classifier for training to screen the calibration object point cloud data and the tooth point cloud data.
The invention also discloses an oral cavity modeling device based on monocular vision, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to realize the steps of the method.
The invention also discloses a computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the method as set forth in any one of the above.
The invention discloses an oral cavity modeling method, an oral cavity modeling system and a storage medium based on monocular vision, wherein a calibration object is attached to the front surface of gum, then a monocular camera is adopted to obtain an oral cavity image with the calibration object, first point cloud data containing characteristics of the oral cavity and the calibration object are generated, then outliers and interference information of the first point cloud data are removed, the first point cloud data are separated to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data, the calibration object point cloud data are compared with a pre-stored actual model of the calibration object to determine a scaling factor of the calibration object, and a tooth three-dimensional model is obtained by reconstructing the tooth point cloud data according to the scaling factor. The whole intraoral scanning process can be completed only by depending on a single-sided camera, and the problems that the cost of the conventional intraoral scanner for three-dimensional reconstruction of teeth in an oral cavity by depending on binocular reconstruction or RGBD (red green blue direct) mode is high and miniaturization is difficult are solved.
Drawings
Fig. 1 is a schematic step diagram of a monocular vision-based oral modeling method according to an embodiment of the present invention.
Fig. 2 is a schematic flowchart of step S5 according to an embodiment of the present invention.
Fig. 3 is a schematic diagram illustrating sampling of the true value of the calibration object according to an embodiment of the present invention.
Fig. 4 is a schematic diagram after plane regularization constraint disclosed in an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. It should be apparent that the described embodiments are only some of the embodiments of the present invention, and not all of them. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the invention without any inventive step, are within the scope of protection of the invention.
In the description of the present invention, it is to be understood that the terms "first", "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood according to specific situations by those of ordinary skill in the art.
In the present invention, unless otherwise expressly stated or limited, "above" or "below" a first feature means that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact with each other via another feature therebetween. Also, the first feature "on," "above" and "over" the second feature may include the first feature being directly above and obliquely above the second feature, or simply indicating that the first feature is at a higher level than the second feature. A first feature being "under," "below," and "beneath" a second feature includes the first feature being directly under and obliquely below the second feature, or simply meaning that the first feature is at a lesser elevation than the second feature.
The intraoral scanning three-dimensional reconstruction technology can carry out three-dimensional reconstruction on teeth in the oral cavity in a monocular reconstruction mode to obtain a digital oral cavity three-dimensional model. However, due to the limitation of monocular three-dimensional reconstruction itself, it is often difficult to determine the actual scale of the model from only one picture. Meanwhile, in order to reduce the cost and avoid using an additional sensor to complete the restoration work of the scale factor, in order to achieve the requirement, the invention obtains the scale scaling factor of the whole model by comparing the three-dimensional model of the calibration object with the physical size of the actual calibration object through a method for measuring the three-dimensional model of the calibration object. As shown in fig. 1, the oral modeling method based on monocular vision may specifically include the following steps:
step S1, acquiring an image of the oral cavity with the calibration object attached to the front surface of the gum by the monocular camera, and generating first point cloud data including characteristics of the oral cavity and the calibration object.
The calibration object can adopt any geometric body with three-dimensional scale data known in advance, such as a cuboid, an ellipsoid and other irregular objects. Before scanning, the calibration object is attached to the front surface of the gum without shielding the tooth characteristics, and then the monocular camera is used for scanning to obtain the oral cavity image containing the calibration object. In this embodiment, a rectangular parallelepiped is preferably used as the calibration object, and the rectangular parallelepiped calibration object is attached to the front surface of the gum to participate in the whole subsequent modeling process.
Step S2, classifying the first point cloud data through a first neural network classifier to form a first point type and a second point type, wherein the first point type is a point cloud attached to the surface of a tooth, a gum or a calibration object, and the second point type is a discrete point cloud.
In this embodiment, the system acquires and labels first point cloud data serving as a sample, labels point clouds attached to teeth, gums or the surface of a calibration object as first-class points, labels discrete point clouds as second-class points, and sends the labeled first-class points and second-class points to a first neural network classifier for training to screen the first-class points.
In particular, since it is difficult to ensure that only teeth are present in the scene during the dental scan, a portion of the inner wall of the oral cavity is also included in the model. While the muscles in the oral cavity have irregular motions, the point cloud formed by the part of the contents is usually wrong and can cause serious interference to the later dense reconstruction. Therefore, it is necessary for the sparse point cloud to first perform a round of cleaning so that the correct points remain, and erroneous or useless points are removed. Therefore, after the sparse reconstruction is completed, the point cloud is manually marked, the point cloud with the teeth attached to the tooth surface is regarded as a type A, namely a first type point, and the discrete point cloud is regarded as a type B, namely a second type point. The two types of data are sent to a three-dimensional point cloud classification model, namely a first neural network classifier, for training, and the model has the capability of sensing outliers after being trained.
In this embodiment, the step S2 further includes:
step S21, each feature point in the first point cloud data as a sample is determined, and if the feature point is larger than the surrounding area, the feature point is retained, otherwise, the adjacent feature point on the right side is used as a maximum candidate point.
And step S22, searching rightward by adopting a monotone increasing mode until finding a characteristic point of I [ I ] > I [ I +1] meeting the ith point, and if I < (W-1), keeping the point. Where w is the total number of elements and i is traversed to ensure that the boundary is not crossed.
Step S23, calculating the quality of the characteristic point
If the point quality is greater than a certain threshold, the point is retained and multiplied by the contribution weight of surrounding points to the point, otherwise the point quality is set to zero. Wherein the weight of the contribution of the surrounding point to the point may be given by neural network regression.
And step S24, judging whether the feature point of the adjacent position of the point and the previous and next frames has redundancy, if so, deleting the feature point, otherwise, keeping and marking the feature point as a first type point.
In this embodiment, the filtering rule may include: the image comprises the quantity of the characteristic points, the quality of the image comprising the characteristic points and the redundancy with the surrounding neighborhood image, and the judgment sequence is executed from front to back. Specifically, first, a single image is subjected to feature extraction, and in consideration of the characteristics of reflection and absorption of light of teeth themselves, a common feature extraction method such as SIFT cannot be used directly, and high-dimensional features of surfaces and contours should be extracted. It is subjected to non-maximum suppression. In a preferred embodiment, the specific algorithm is as follows:
firstly, according to line 3, judging whether current characteristic point I [ I ] is greater than its peripheral neighborhood, if it is in accordance with condition, retaining. If the characteristic I [ I ] does not satisfy the judgment condition of the 3 rd row, the right adjacent I [ I +1] is taken as a maximum candidate and corresponds to the 7 th row. And searching rightward in a monotonically increasing mode until a characteristic point meeting I [ I ] > I [ I +1] is found, and if I < ═ W-1, keeping the point.
The quality of the spot is then judged:
if the point is larger than a certain threshold value, the point is reserved and multiplied by the contribution weight of the surrounding points to the point, otherwise, the point is directly set to 0. And finally, judging whether the feature point is redundant with the feature point at the adjacent position of the point and the previous and next frames so as to avoid generating the same feature due to overlong retention time.
Step S3, traversing each frame of oral cavity image, deleting the feature points and constraint relations of the second type points in each frame of image, taking the feature points in all the remaining frame images as candidate feature points of the first type points, deleting the whole frame of image which does not contain the first type points, and acquiring second point cloud data consisting of the first type points. When a new point cloud enters, the point cloud is classified by using the three-dimensional point cloud classification model which is trained in the step S2, namely the first neural network classifier, so that the A-type points pass through, the B-type points are prevented from participating in subsequent calculation, all images are traversed, and the feature points with the B-type points and the constraint relation are deleted. All the remaining frames only leave the class a points as candidate feature points. And if the image does not contain the A-type points, deleting the frame image.
And step S4, screening and classifying the second point cloud data through a second neural network classifier, and respectively acquiring tooth point cloud data, calibration object point cloud data and gum point cloud data.
The step S4 includes: the calibration object is a cuboid calibration object, second point cloud data serving as a sample are obtained, different labels are respectively carried out on point clouds belonging to the cuboid calibration object and teeth in the second point cloud data, and the second point cloud data after being labeled are sent to a second neural network classifier to be trained and used for screening the point cloud data of the calibration object and the point cloud data of the teeth. Specifically, after a complete model is obtained, the point cloud in the cuboid calibration block is regarded as a target through manual marking for training, so that the point cloud has the capability of distinguishing calibration objects.
And step S5, comparing the point cloud data of the calibration object with a pre-stored actual model of the calibration object to determine a scaling factor of the calibration object, and reconstructing the point cloud data of the teeth according to the scaling factor to obtain a three-dimensional model of the teeth.
Due to the existence of modeling errors and segmentation errors, the point cloud obtained at the moment cannot be directly used for scale calculation, and a series of processing is needed to perfect the process. In this embodiment, as shown in fig. 2, the step S5 includes:
step S51, performing three-dimensional model uniform sampling on the calibration object alone to obtain first sampling data, which is a point cloud obtained after uniform sampling of the calibration object as shown in fig. 3. In this embodiment, an original calibration object STL model generated by Rhino modeling and a calibration object three-dimensional model generated by monocular three-dimensional reconstruction may be uniformly sampled. The uniform sampling step mainly comprises the following steps:
in step S511, an expected number N of sampling points that is much larger than the number of triangle surfaces of the three-dimensional model file is set.
Step S512, taking out a triangular surface in sequence each time and recording the triangular surface as Triangle, wherein the coordinates of three vertexes are respectively recorded as v 1 、v 2 And v 3 。
Step S513, generating random number r according to continuous uniform distribution 1 And r 2 Meanwhile, the weight a, b and c of each vertex can be calculated according to the random number. Wherein
The sampling generates the coordinates v 'of the new point, where v' is a v 1 +b*v 2 +c*v 3 。
And step S514, when the number of sampled points is less than the number N to be sampled, repeating the steps S512-S513.
Step S52, performing plane extraction on the first sampling data, and acquiring point data of four planes of the calibration object contacting with each other in pairs as point cloud data of adjacent surfaces. Specifically, the plane extraction is performed on the first sample data acquired in step S51, and the first sample data is divided into A, B, C, D total four planes, and the step may specifically include the following steps:
step S521, randomly selecting non-collinear 3 points from all points of the point cloud to construct a plane equation
Let these three points denote p, respectively 1 、p 2 、p 3 Wherein
Wherein d may be an intercept, and p1 may be a point randomly selected from p1 p2 p 3.
Step S522, calculating the distance between the rest points and the plane equation, wherein the distance smaller than the threshold value can be used as the inner point of the plane model, and p is used i Representing other points in the point cloud except the three points, the distance formula can be expressed as:
in step S523, the above steps S521 and S522 are repeated for a first predetermined number of times, where the first predetermined number of times may be 1000 times, and the plane equation with the largest number of interior points is taken as the surface extracted this time.
In step S524, repeating step S523 a second predetermined number of times, where the second predetermined number of times may be 4, and extracting four surfaces a, B, C, and D, respectively. Then, the ab surface and the cd surface can be separated according to the relation between the two surfaces which are parallel and coplanar. So far, the fitted plane equations and the inner points of the planes can be obtained.
And step S53, adding plane regularization constraint to the first sampling data to obtain second sampling data, wherein the plane regularization constraint is that the distance from each surface association point of the adjacent surface point cloud data to the current surface is minimum, the included angle between two surfaces containing less point cloud data approaches to zero, and the included angle between two surfaces containing less point cloud data and two surfaces containing more point cloud data approaches to 90 degrees.
Specifically, a plane regularization constraint is added to the point cloud extracted in step S53. There are two error terms, one is the point-to-plane distance and the other is the face-to-face angle relationship. Not only the minimum distance from the point associated with each surface to the current surface is ensured, but also the included angle between the surface a and the surface B is ensured to be close to zero, and the included angles between the surface a, the surface C and the surface C, the surface D are close to 90 degrees, as shown in fig. 4.
Step S54, comparing the second sampling data with the point cloud data of the calibration object to determine the scaling factor of the calibration object.
Specifically, according to the result of the foregoing steps, the scaling factor s of the calibration object obtained by monocular reconstruction can be estimated 0 ，s 0 Is the ratio of the actual length of the calibration object to the estimated length after reconstruction. Then [0.8s ] 0 ,1.2s 0 ]And performing multi-step search in the range, wherein the specific steps are as follows:
let the step length of the first round of search be 0.1s 0 . The standard point cloud is cube gt The reconstructed point cloud is cube i . Traverse cube i Every point in it is found in cube gt The nearest point in the area is set as a corresponding point. Solving for translation and rotation using an iterative approach such that cube i The square sum of the distances from the upper point to the tangent plane of the corresponding point is minimum, and the integral error at the moment is recorded as error i 。
Repeating the steps for a preset number of times, wherein 20 times can be selected until the interval is searched, and at the moment, s 'with the minimum error can be obtained' 0 . The step length is reduced by 0.1 time, and the operation is iterated for 3 times, so that a relatively accurate scale scaling factor can be obtained.
The embodiment is characterized in that a calibration object is attached to the front surface of a gum, then a monocular camera is adopted to obtain an oral cavity image with the calibration object, first point cloud data containing characteristics of the oral cavity and the calibration object are generated, then outliers and interference information are removed from the first point cloud data, the first point cloud data are separated to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data, the calibration object point cloud data are compared with a pre-stored actual calibration object model to determine a scaling factor of the calibration object, and a tooth three-dimensional model is obtained by reconstructing the tooth point cloud data according to the scaling factor. The whole intraoral scanning process can be completed only by depending on a single-sided camera, and the problems that the cost of the conventional intraoral scanner for three-dimensional reconstruction of teeth in an oral cavity by depending on binocular reconstruction or RGBD (red green blue direct) mode is high and miniaturization is difficult are solved.
In other embodiments, there is also disclosed a monocular vision based oral scanning system comprising a monocular camera and a controller coupled to the monocular camera, the controller configured to: acquiring an oral cavity image with a calibration object through a monocular camera, wherein the calibration object is attached to the front surface of the gum, and first point cloud data containing characteristics of the oral cavity and the calibration object are generated; classifying the first point cloud data through a first neural network classifier to form a first point type and a second point type, wherein the first point type is a point cloud attached to the surface of a tooth, a gum or a calibration object, and the second point type is a discrete point cloud; traversing each frame of oral cavity image, deleting the characteristic points and constraint relations in each frame of image as second-class points, taking the characteristic points in all the remaining frame of image as candidate characteristic points of the first-class points, deleting the whole frame of image which does not contain the first-class points, and acquiring second point cloud data consisting of the first-class points; screening and classifying the second point cloud data through a second neural network classifier, and respectively obtaining tooth point cloud data, calibration object point cloud data and gum point cloud data; and comparing the point cloud data of the calibration object with a pre-stored actual model of the calibration object to determine a scaling factor of the calibration object, and reconstructing the point cloud data of the teeth according to the scaling factor to obtain a three-dimensional model of the teeth.
In this embodiment, the controller is further configured to label the acquired first point cloud data as a sample, label the point cloud attached to the tooth, gum or surface of the calibration object as a first type of point, label the discrete point cloud as a second type of point, and send the labeled first type of point and second type of point to the first neural network classifier for training to screen the first type of point.
In this embodiment, the controller is further configured to obtain second point cloud data serving as a sample, perform different labeling on point clouds belonging to a cuboid calibration object and teeth in the second point cloud data, and send the labeled second point cloud data to a second neural network classifier for training to screen the calibration object point cloud data and the tooth point cloud data.
The specific functions of the oral cavity scanning system based on monocular vision correspond to the oral cavity scanning method based on monocular vision disclosed in the previous embodiments one to one, so detailed description is omitted here, and specific reference may be made to each embodiment of the oral cavity scanning method based on monocular vision disclosed in the previous embodiments. It should be noted that, in the present specification, each embodiment is described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other.
In other embodiments, there is also provided a monocular vision based oral scanning device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method for monocular vision based oral scanning as described in the above embodiments when executing the computer program.
Wherein the monocular vision based oral scanning device may include, but is not limited to, a processor, a memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of a monocular vision based oral scanning device, and does not constitute a limitation of the monocular vision based oral scanning device, and may include more or less components than those shown, or combine some components, or different components, for example, the monocular vision based oral scanning device may further include an input-output device, a network access device, a bus, etc.
The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being the control center of the monocular vision based intraoral scanning appliance apparatus, various interfaces and lines connecting the various parts of the entire monocular vision based intraoral scanning appliance apparatus.
The memory may be used to store the computer programs and/or modules, and the processor may implement the various functions of the apparatus for monocular vision based oral scanning by running or executing the computer programs and/or modules stored in the memory and invoking the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like, and the memory may include a high speed random access memory, and may further include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
The monocular vision based oral scanning device, if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be implemented by a computer program, which can be stored in a computer-readable storage medium, and the computer program can implement the above-mentioned steps for the monocular vision-based oral cavity scanning method embodiment when being executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
In summary, the above-mentioned embodiments are only preferred embodiments of the present invention, and all equivalent changes and modifications made in the claims of the present invention should be covered by the claims of the present invention.
Claims (10)
1. An oral modeling method based on monocular vision, characterized by comprising the following steps:
s1, acquiring an oral cavity image with a calibration object through the monocular camera, wherein the calibration object is attached to the front surface of the gum, and first point cloud data containing characteristics of the oral cavity and the calibration object are generated;
s2, classifying the first point cloud data through a first neural network classifier to form a first point type and a second point type, wherein the first point type is a point cloud attached to the surface of a tooth, a gum or a calibration object, and the second point type is a discrete point cloud;
s3, traversing each frame of oral cavity image, deleting the feature points and constraint relations in each frame of image as second-class points, taking the feature points in all the remaining frame images as candidate feature points of the first-class points, deleting the whole frame of image which does not contain the first-class points, and acquiring second point cloud data consisting of the first-class points;
s4, screening and classifying the second point cloud data through a second neural network classifier, and respectively acquiring tooth point cloud data, calibration object point cloud data and gum point cloud data;
and S5, comparing the point cloud data of the calibration object with a pre-stored actual model of the calibration object to determine a scaling factor of the calibration object, and reconstructing the point cloud data of the teeth according to the scaling factor to obtain a three-dimensional model of the teeth.
2. The monocular vision based oral modeling method of claim 2, wherein said step S2 comprises:
the method comprises the steps of obtaining and marking first point cloud data serving as a sample, marking point clouds attached to teeth, gum or the surface of a calibration object as first-class points, marking discrete point clouds as second-class points, and sending the marked first-class points and second-class points into a first neural network classifier for training to screen the first-class points.
3. The monocular vision based oral modeling method of claim 2, wherein said step S2 further comprises:
judging each feature point in the first point cloud data serving as the sample respectively, if the feature point is larger than the surrounding field, reserving the feature point, and otherwise, taking the adjacent feature point on the right side as a maximum candidate point;
searching rightwards by adopting a monotone increasing mode until a characteristic point meeting I [ I ] > I [ I +1] is found, and if I < ═ W-1, reserving the point;
calculating the characteristic point quality
If the point quality is greater than a certain threshold value, retaining the point and multiplying by the contribution weight of the surrounding points to the point, otherwise setting the point quality to zero;
and judging whether the feature point has redundancy with the feature point at the position adjacent to the previous frame and the next frame, if so, deleting the feature point, and otherwise, keeping and marking the feature point as a first type point.
4. The monocular vision based oral modeling method of claim 3, wherein said step S4 comprises:
the calibration object is a cuboid calibration object, second point cloud data serving as a sample are obtained, different labels are respectively carried out on point clouds belonging to the cuboid calibration object and teeth in the second point cloud data, and the second point cloud data after being labeled are sent to a second neural network classifier to be trained and used for screening the point cloud data of the calibration object and the point cloud data of the teeth.
5. The monocular vision based oral modeling method of claim 4, wherein said step S5 comprises:
uniformly sampling a three-dimensional model of a calibration object to obtain first sampling data;
performing plane extraction on the first sampling data, and acquiring point data of four planes of the calibration object which are contacted with each other in pairs as point cloud data of adjacent surfaces;
adding a plane regularization constraint to the first sampling data to obtain second sampling data, wherein the plane regularization constraint is that the distance from each surface association point of the adjacent surface point cloud data to the current surface is minimum, the included angle of two surfaces containing less point cloud data approaches to zero, and the included angle of two surfaces containing less point cloud data and two surfaces containing more point cloud data approaches to 90 degrees;
and comparing the second sampling data with the point cloud data of the calibration object to determine a scaling factor of the calibration object.
6. A monocular vision based oral scanning system comprising a monocular camera and a controller connected to the monocular camera, the controller configured to:
acquiring an oral cavity image with a calibration object through a monocular camera, wherein the calibration object is attached to the front surface of the gum, and first point cloud data containing characteristics of the oral cavity and the calibration object are generated;
classifying the first point cloud data through a first neural network classifier to form a first point type and a second point type, wherein the first point type is a point cloud attached to the surface of a tooth, a gum or a calibration object, and the second point type is a discrete point cloud;
traversing each frame of oral cavity image, deleting the characteristic points and constraint relations in each frame of image as second-class points, taking the characteristic points in all the remaining frame of image as candidate characteristic points of the first-class points, deleting the whole frame of image which does not contain the first-class points, and acquiring second point cloud data consisting of the first-class points;
screening and classifying the second point cloud data through a second neural network classifier, and respectively acquiring tooth point cloud data, calibration object point cloud data and gum point cloud data;
and comparing the point cloud data of the calibration object with a pre-stored actual model of the calibration object to determine a scaling factor of the calibration object, and reconstructing the point cloud data of the teeth according to the scaling factor to obtain a three-dimensional model of the teeth.
7. The monocular vision based oral scanning system of claim 6, wherein the controller is further configured to label the acquired first point cloud data as a sample, label the point cloud attached to the tooth, gum or surface of the calibration object as a first type of point, label the discrete point cloud as a second type of point, and send the labeled first type of point and second type of point to a first neural network classifier for training in screening the first type of point.
8. The monocular vision based oral modeling system of claim 7, wherein the controller is further configured to obtain second point cloud data as a sample, perform different labeling on point clouds belonging to a cuboid landmark and a tooth in the second point cloud data, and send the labeled second point cloud data to a second neural network classifier for training for screening the landmark point cloud data and the tooth point cloud data.
9. A monocular vision based oral modeling apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: the processor, when executing the computer program, realizes the steps of the method according to any of claims 1-5.
10. A computer-readable storage medium storing a computer program, characterized in that: the computer program realizing the steps of the method according to any of claims 1-5 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210676629.2A CN115100350A (en) | 2022-06-15 | 2022-06-15 | Monocular vision-based oral modeling method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210676629.2A CN115100350A (en) | 2022-06-15 | 2022-06-15 | Monocular vision-based oral modeling method, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115100350A true CN115100350A (en) | 2022-09-23 |
Family
ID=83291532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210676629.2A Pending CN115100350A (en) | 2022-06-15 | 2022-06-15 | Monocular vision-based oral modeling method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115100350A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115375903A (en) * | 2022-10-27 | 2022-11-22 | 天津大学 | Method and system for obtaining reconstruction data for reconstructing teeth |
-
2022
- 2022-06-15 CN CN202210676629.2A patent/CN115100350A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115375903A (en) * | 2022-10-27 | 2022-11-22 | 天津大学 | Method and system for obtaining reconstruction data for reconstructing teeth |
| CN115375903B (en) * | 2022-10-27 | 2023-01-17 | 天津大学 | Method and system for obtaining reconstruction data for reconstructing teeth |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11928832B2 (en) | Intraoral scanning system that removes excessive materials from intraoral scans | |
| US20210322136A1 (en) | Automated orthodontic treatment planning using deep learning | |
| CN110087577B (en) | Method and system for removing braces from dentition grid | |
| CN101697233B (en) | Structured light-based three-dimensional object surface reconstruction method | |
| US9007602B2 (en) | Three-dimensional measurement apparatus, three-dimensional measurement method, and computer-readable medium storing control program | |
| JP2019523064A5 (en) | ||
| CN109949347B (en) | Human body tracking method, device, system, electronic equipment and storage medium | |
| KR20190037241A (en) | METHOD AND SYSTEM FOR REMOVING DIETARY METAL BRASCE | |
| US11250580B2 (en) | Method, system and computer readable storage media for registering intraoral measurements | |
| JP6573907B2 (en) | System, method, apparatus, and computer readable storage medium for collecting color information about 3D scanned objects | |
| TW201118791A (en) | System and method for obtaining camera parameters from a plurality of images, and computer program products thereof | |
| Silvester et al. | A critical assessment of the potential for structure‐from‐motion photogrammetry to produce high fidelity 3D dental models | |
| CA2623053A1 (en) | Artifact mitigation in three-dimensional imaging | |
| WO2020119467A1 (en) | High-precision dense depth image generation method and device | |
| WO2020263987A1 (en) | Processing digital dental impression | |
| Mirzaalian Dastjerdi et al. | Measuring surface area of skin lesions with 2D and 3D algorithms | |
| EP4149385A1 (en) | Intra-oral scanning device with active delete of unwanted scanned items | |
| JPH11242745A (en) | Method for measuring and processing facial image | |
| JP2019530059A (en) | Method for independently processing multiple target areas | |
| Slizewski et al. | Surface scanning of anthropological specimens: nominal-actual comparison with low cost laser scanner and high end fringe light projection surface scanning systems: Oberflächenscannen anthropologischer Objekte: Soll-Ist Vergleiche mit einem niedrigpreisigen Laserscanner und hochpreisigen Streifenlicht Scanner Systemen | |
| CN112106109A (en) | Intraoral scanning with surface differentiation | |
| CN110738703A (en) | Positioning method and device, terminal and storage medium | |
| JP2001243468A (en) | Device, method, and metdium for three-dimensional modeling, and device, method, and medium for three- dimensional shape data recording | |
| CN115100350A (en) | Monocular vision-based oral modeling method, system and storage medium | |
| JP2011095131A (en) | Image processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |