CN114283236A - Method, device and storage medium for oral cavity scanning by using smart phone - Google Patents
Method, device and storage medium for oral cavity scanning by using smart phone Download PDFInfo
- Publication number
- CN114283236A CN114283236A CN202111545540.4A CN202111545540A CN114283236A CN 114283236 A CN114283236 A CN 114283236A CN 202111545540 A CN202111545540 A CN 202111545540A CN 114283236 A CN114283236 A CN 114283236A
- Authority
- CN
- China
- Prior art keywords
- image
- images
- oral cavity
- module
- group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Dental Tools And Instruments Or Auxiliary Dental Instruments (AREA)
Abstract
The invention discloses a method, a device and a storage medium for oral cavity scanning by using a smart phone.A front end of a camera lens of the smart phone is connected with an external probe which can be used for transmitting an image of teeth in an oral cavity, a series of images with different depth of field in the oral cavity of a patient are obtained by high-speed shooting while the offset of the camera lens of the smart phone is changed, a single-view depth map is formed after the collected images are processed, the depth maps with multiple viewing angles are converted into point cloud data and then spliced and gridded to form a final three-dimensional model, so that a real image of the teeth in the oral cavity of the patient is transmitted into the smart phone to be restored into the oral cavity model, the cost of oral cavity scanning business is greatly reduced, and the efficiency is improved.
Description
Technical Field
The invention belongs to the field of auxiliary design of digital intelligent equipment for oral modeling, and particularly relates to a method and a device for oral scanning by using a smart phone and a storage medium.
Background
In the clinical oral diagnosis and treatment and repair process, the method for obtaining the three-dimensional digital model of the interior of the oral cavity through three-dimensional scanning mainly comprises two types of extraoral scanning and intraoral scanning. The extraoral scanning mode requires that a doctor firstly takes an impression of an oral cavity to obtain a plaster model of a tooth, and then scans the plaster model of the tooth by using three-dimensional scanning equipment to obtain a three-dimensional digital model of the tooth; the intraoral scanning mode is that the intraoral scanner is extended into the oral cavity to directly scan the teeth so as to obtain the three-dimensional digital model of the teeth, and the intraoral scanning mode has the advantages of simple operation, high efficiency and high measurement speed, thereby saving the time of chair-side operation of doctors, avoiding errors caused by mold making and mold overturning due to no need of manual impression, and having higher measurement precision. However, the existing oral cavity scanners are all composed of an oral cavity scanner and a huge host station connected with the oral cavity scanner. During operation, a doctor holds the scanner by hand and puts the scanner into the oral cavity of a patient, the scanner camera continuously shoots and transmits the image to the host station, and real-time 3D modeling is carried out by host station software. When the mouth is opened, the mouth and the tongue are uncomfortable and constantly move, and doctors need to observe and adjust at any time and communicate with patients. Meanwhile, in order to enable the scanner to shoot images at a required position and not to frequently collide or push against other parts of the oral cavity, the sight of a doctor needs to continuously move between the oral cavity and the screens of the host stations beside the oral cavity, so that the scanning efficiency is reduced, and in addition, the large host stations cause the whole oral scanner system to occupy a large amount of limited diagnosis and treatment space and be high in manufacturing cost.
Disclosure of Invention
The invention provides a method for scanning oral cavity by using a smart phone aiming at the defects in the prior art, wherein the front end of a camera lens of the smart phone is connected with an external probe which can be used for transmitting the tooth image in the oral cavity, and the method comprises the following steps:
s1, calibrating a camera module of the smart phone, controlling the camera module to shoot the calibration plate at different distances, and calibrating the moving distance of a focal plane in the focusing process;
s2, acquiring a first focal length of the camera module when the camera module forms the clearest image on the surface of a tooth, and zooming the first focal length to obtain a focal length range capable of covering the surface of the tooth;
s3, controlling the camera module to take a picture at a high speed and zoom quickly in the focal range to obtain a group of continuous initial images in the oral cavity, and capturing different depth-of-field images in the group of continuous initial images as a first image group according to the focal range;
and S4, aligning each frame image in the first image group, sequentially matching the frame images with the previous image by taking the last image as a reference to calculate a zoom factor, zooming each frame image in the first image group according to the zoom factor, superposing to form a single-view depth map, and calling an intelligent mobile phone inertia measurement module to acquire corresponding pose information when the group of depth maps is acquired.
Preferably, the method for oral cavity scanning with a smartphone further includes the following steps:
and S5, converting the acquired single-view depth maps into point cloud data, and iterating by using the pose change translation and rotation matrix between the two frames of images and scale information obtained according to the calibration result as initial values to minimize the alignment error of the two point clouds and splicing to form the oral cavity tooth three-dimensional model.
Preferably, the step S4 specifically includes:
s41, sequentially selecting front and rear frames of images of the first image group, searching by using an SIFT algorithm to obtain respective feature points, solving the transformation relation of the two sets of feature points by using a RANSAC algorithm, solving a corresponding homography matrix, and transforming the second image by using the solved matrix to obtain a transformed second image group consisting of images with near-far small effect eliminated;
s42, overlapping the images of each frame in the second image group together for Laplace transform, selecting the largest image in the response values under the same pixel coordinate as the focal plane corresponding to the pixel, and obtaining the depth corresponding to each pixel according to the depth value of the focal plane;
and S43, sequentially acquiring the pixel coordinates of each pixel when the contrast ratio is maximum from front to back, recording the pixel coordinates to form the visual angle depth map, and calling the smart phone inertial measurement module to acquire the pose information corresponding to the acquired depth map.
Preferably, the step S41 specifically includes:
s411, searching front and back frames of images in the first image group by using an SIFT algorithm to obtain respective feature points;
s412, randomly selecting a plurality of groups of matched feature points to calculate a homography matrix, calculating errors of other matched feature points after homography matrix transformation, obtaining the number of the feature points with the errors smaller than a threshold value, and obtaining the homography matrix with the largest number of the feature points with the errors smaller than the threshold value;
s413, transforming the second image by using the solved homography matrix to obtain a transformed image for eliminating the near-far effect;
and S414, continuously adopting the steps S31 to S33 to sequentially transform the front frame image and the back frame image in the first image group to obtain a transformed image with the near-far-small effect eliminated, and forming a second image group.
The invention also discloses an oral cavity scanning device, which comprises an external probe with an optical image transmission component inside and a mobile terminal which is detachably connected with the external probe, wherein the mobile terminal is provided with a camera module capable of acquiring the transmission image of the external probe, an inertia measurement module capable of acquiring the pose state of the mobile terminal and a control module, and the control module specifically comprises: the calibration module is used for controlling the camera module to shoot the calibration plate at different distances and calibrating the moving distance of the focal plane in the focusing process; the focal length range acquisition module is used for acquiring a first focal length of the camera module when the camera module forms the clearest image on the surface of a tooth, and zooming the first focal length to acquire a focal length range capable of covering the surface of the tooth; the first image group module is used for controlling the camera module to take a picture at a high speed and simultaneously zoom quickly in the focal range to obtain a group of continuous initial images in the oral cavity, and capturing different depth-of-field images in the group of continuous initial images as a first image group according to the focal range; and the single-view depth map generation module is used for aligning each frame of image in the first image group, then taking the last image as a reference, sequentially matching the last image with the previous image to calculate a zoom factor, zooming each frame of image in the first image group according to the zoom factor, then overlapping to form a single-view depth map, and calling the intelligent mobile phone inertia measurement module to acquire corresponding pose information when the group of depth maps is acquired.
Preferably, the control module further comprises: and the three-dimensional model generation module is used for converting the acquired single-view depth maps into point cloud data, and iterating by using the pose change translation and rotation matrix between the two frames of images and scale information obtained according to the calibration result as an initial value to minimize the alignment error of the two point clouds and form the oral cavity tooth three-dimensional model by splicing.
Preferably, the single-view depth map generating module specifically includes: the second image group acquisition module is used for sequentially selecting two frames of images before and after the first image group, searching the images by using an SIFT algorithm to obtain respective feature points, solving the transformation relation of the two groups of feature points by using an RANSAC algorithm, solving a corresponding homography matrix, and transforming the second image by using the solved matrix to obtain a transformed second image group consisting of the images with the near-far effect eliminated; the depth acquisition module is used for superposing the frames of images in the second image group together to perform Laplace transform, selecting the largest image in response values under the same pixel coordinate as a focal plane corresponding to the pixel, and acquiring the depth corresponding to the pixel according to the depth value of the focal plane; and the depth map module is used for sequentially acquiring the pixel coordinates of each pixel when the contrast is maximum from front to back, recording the pixel coordinates to form the visual angle depth map, and calling the smart phone inertial measurement module to acquire the pose information corresponding to the acquired depth map.
Preferably, the second image group acquiring module specifically includes: the feature point module is used for searching front and rear frames of images in the first image group by using an SIFT algorithm to obtain respective feature points; the homography matrix acquisition module is used for randomly selecting a plurality of groups of matched characteristic points to calculate a homography matrix, calculating errors of other matched characteristic point pairs after homography matrix transformation, acquiring the number of the characteristic point pairs with the errors smaller than a threshold value, and acquiring the homography matrix with the largest number of the characteristic point pairs with the errors smaller than the threshold value; the transformed image acquisition module is used for transforming the second image by using the solved homography matrix to obtain a transformed image for eliminating the near-far effect; and the second image generation module is used for continuously transforming the front frame image and the rear frame image in the first image group to obtain a transformed image for eliminating the near-far effect and form a second image group.
The invention also discloses a device for oral cavity scanning by using the smart phone, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to realize the steps of the method for oral cavity scanning by using the smart phone.
The invention also discloses a computer readable storage medium storing a computer program which, when executed by a processor, implements the method steps for oral scanning with a smartphone as described above.
The invention discloses a method and a device for carrying out oral scanning by using a smart phone, which are characterized in that a lens module of the smart phone is used as an oral scanning device to carry out three-dimensional reconstruction on teeth in an oral cavity, a series of images with different depths of field in the oral cavity of a patient are obtained by high-speed photography, the collected data are deblurred, filtered and aligned, the last image is used as a calibration reference, each deblurred image is matched with the calibration, the zoom ratio between the front image and the rear image is calculated, then the images are zoomed according to the zoom ratio to eliminate the near-far effect, the images are overlapped together to form a single-view depth image after being subjected to Laplace transform filtering, finally the depth images of a plurality of view angles are obtained to form a depth image, point cloud registration, splicing and gridding are carried out to form a final three-dimensional model. Therefore, the real image in the internal environment of the oral cavity of the patient is transmitted into the smart phone, the real model of the oral cavity is restored, the cost of the oral cavity scanning business is greatly reduced, and the efficiency is improved. The problem of present oral cavity scanner all have the oral cavity scanner and with the huge host computer station constitution of oral cavity scanner connection is solved, the doctor sight need constantly move between oral cavity and body side host computer station screen when the operation, cause scanning efficiency to reduce, huge host computer station leads to whole oral cavity scanner system to occupy a large amount of limited diagnosis and treatment space and the cost is high in addition.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
Fig. 1 is a flowchart illustrating a method for performing oral cavity scanning with a smart phone according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a smart phone according to an embodiment of the present invention connected to an external probe;
fig. 3 is a flowchart illustrating a step S4 according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating the effect of feature matching according to an embodiment of the present invention;
fig. 5 is a flowchart illustrating the step S41 according to an embodiment of the present invention;
FIG. 6 is a schematic view of a single perspective point cloud according to an embodiment of the present disclosure;
FIG. 7 is a digital model of an oral cavity after multi-view stitching according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an oral cavity scanning device according to an embodiment of the disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the invention without any inventive step, are within the scope of protection of the invention.
In the description of the present invention, it is to be understood that the terms "first", "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the present invention, unless otherwise expressly stated or limited, "above" or "below" a first feature means that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact with each other via another feature therebetween. Also, the first feature being "on," "above" and "over" the second feature includes the first feature being directly on and obliquely above the second feature, or merely indicating that the first feature is at a higher level than the second feature. A first feature being "under," "below," and "beneath" a second feature includes the first feature being directly under and obliquely below the second feature, or simply meaning that the first feature is at a lesser elevation than the second feature.
The embodiment of the invention discloses a method for scanning oral cavity by using a smart phone, wherein as shown in figure 1, the front end of a camera lens of the smart phone is connected with an external probe which can be used for transmitting an image of teeth in the oral cavity, as shown in figure 2, the method for scanning oral cavity by using the smart phone can specifically comprise the following steps:
and S1, calibrating the camera module of the smart phone, controlling the camera module to shoot the calibration plate at different distances, and calibrating the moving distance of the focal plane in the focusing process.
Specifically, the camera module of the smart phone is calibrated, and the depth value of each focal plane is obtained when the smart phone is focused. The moving distance of each two picture focal planes in the focusing process is calibrated by shooting the calibration plate at different distances, wherein the calibration plate can be selected from a circular array. The method comprises the following steps:
in step S11, the radius of the point spread function is measured using the Hough Circle Transform algorithm. The point spread function acts on the image to generate a fuzzy radius, the part close to the center of the circle is clearer, and the edge of the radius is more fuzzy. For each pattern, the part with the definition between 5% and 95% positions the effective area, and other over-blurred parts are discarded, so that the range and the blur radius generated by the blur can be obtained. On this basis, by setting thresholds for 5% and 95% of the image intensity, the blur radius of the circular border region is obtained.
And step S12, calculating the change of the distance between the centers of the circles to obtain a change curve of the magnification in the focusing process. The method comprises the steps of extracting characteristic points of adjacent frame images, estimating the change relation of the characteristic points, estimating the change generated by the images by using the difference between the characteristic points, calculating the change curve of the images in the focusing process, and reducing the problems of the near and far by the method.
Step S2, obtaining a first focal length of the camera module when forming the clearest image on a tooth surface, and zooming the first focal length to obtain a focal length range that can cover the tooth surface. Specifically, when the lens module of the smart phone can acquire a tooth image through the external probe, the lens module is controlled to continuously adjust the focal length to the focal length of the current tooth which can be clearly displayed on the display screen of the smart phone, and the focal length is zoomed so as to acquire the focal length range which can cover the surface of the current tooth.
And step S3, controlling the camera module to take a picture at a high speed and zoom quickly in the focal range to obtain a group of continuous initial images in the oral cavity, and intercepting different depth-of-field images in the group of continuous initial images as a first image group according to the focal range.
When the inside of the oral cavity is shot, the lens module is controlled to zoom and record the current focal length during shooting, and simultaneously, images with different depths of field in the corresponding oral cavity are recorded. The portable optical lens device is firstly installed at the front end of the mobile phone, and then the device is inserted into the mouth of a patient by a stomatologist and is continuously swept over the teeth of the mouth of the patient according to a certain sequence. When scanning, the mobile phone camera takes pictures at a high speed, so that the phenomenon that continuous frames cannot be matched with each other due to irregular muscle vibration is avoided as much as possible. And changing the focal length of the lens in the shooting process to obtain a group of continuous images in the oral cavity. In this embodiment, the camera is turned on first, and a tooth is aimed, and the focal distance that makes the current tooth sharpest is found by auto-focusing. The focal length is scaled to find a range of focal segments that can cover the current tooth surface, as shown in fig. 4. And then, high-speed photographing is started, and meanwhile, zooming is performed rapidly in the focal section, so that the phenomenon that front and back continuous frames cannot be matched due to irregular shaking of muscles is avoided, and a group of continuous initial images in the oral cavity are obtained. For example, if the phone takes a picture at 120fps, 120 pictures can be taken in 1 second, and then a subset of the set of images can be cropped for subsequent processing based on the calculated focal range for auto-focusing.
And step S4, aligning each frame image in the first image group, then matching the frame image with the previous image in sequence by taking the last image as a reference to calculate a zoom factor, zooming each frame image in the first image group according to the zoom factor, then overlapping to form a single-view depth map, and calling an intelligent mobile phone inertial measurement module to acquire corresponding pose information when the group of depth maps is acquired.
As shown in fig. 3, step S4 specifically includes:
and step S41, sequentially selecting two frames of images before and after the first image group, searching by using SIFT algorithm to obtain respective feature points, solving the transformation relation of the two groups of feature points by using RANSAC algorithm, solving the corresponding homography matrix, and transforming the second image by using the solved matrix to obtain a transformed second image group consisting of the images with the near-far effect eliminated.
In particular, the alignment correction is required for the group of images because the focal plane shift will cause the group of images to appear sharp to blurred, and at the same time, will produce a large-to-small effect. When aligning and correcting, firstly, searching front and back frame images by using an SIFT algorithm to obtain respective feature points; then, solving the transformation relation of the two groups of feature points by using a RANSAC algorithm, and solving a corresponding homography matrix; and finally, transforming the second image by using the solved matrix to eliminate the near-far effect. And smoothing the group of images by using median filtering to eliminate interference caused by noise generated by jitter.
In this embodiment, as shown in fig. 5, the step S41 may further specifically include the following steps.
Step S411, searching front and back frames of images in the first image group by using SIFT algorithm to obtain respective feature points. Because the stability of the handheld shooting mode is poor, the image jitter needs to be eliminated, the near-far effect caused by the lens movement also needs to be eliminated, and the front frame image and the rear frame image are searched by using an SIFT algorithm to obtain respective feature points.
Step S412, a plurality of groups of matched feature points are randomly selected to calculate a homography matrix, the errors of the rest matched feature points after homography matrix transformation are calculated, the number of the feature points with the errors smaller than a threshold value is obtained, and the homography matrix with the largest number of the feature points with the errors smaller than the threshold value is obtained. Specifically, the RANSAC algorithm is used for solving the transformation relation of the two groups of feature points, namely solving the corresponding homography matrix. The RANSAC comprises the following specific steps: first, 4 sets of matched feature points are selected and their homography matrices are calculated. And then calculating the error of the other matched characteristic point pairs after homography transformation, and recording the number of the characteristic point pairs with the error smaller than a threshold value. And finally, after the iteration is finished, the homography matrix with the largest number of the feature point pairs with the error smaller than the threshold value is obtained.
In this embodiment, the hamming distance between descriptors is used as a metric, two groups of feature points are searched by combining approximate nearest neighbor to obtain the matching relationship between the two groups of feature points, and a homography matrix is calculated by randomly selecting four pairs of matched feature points. The Hamming distance formula is shown below:
wherein a and b represent binary descriptors of each group of feature points.
In this embodiment, the homography matrix is calculated as follows:
there is a pair of matched feature points (u1, v1), (u2, v2) that satisfy a homographic transformation relationship:
The case of extension to four pairs of matching points may be:
the vector H is the homography transformation matrix H.
In step S413, the second image is transformed by using the solved homography matrix, and a transformed image with the near-far effect eliminated is obtained. Specifically, the second image I is transformed by using the solved matrix H, so as to obtain a picture I 'with the near-far small effect eliminated, wherein I' is HI.
And step S414, continuing to adopt the foregoing steps S411 to S413 to sequentially transform the two frames of images in the first image group to obtain transformed images with the near-far-small effect eliminated, and forming a second image group. In addition, the group of images are smoothed by using median filtering, so that interference caused by noise caused by jitter is eliminated. And the pixel value of each pixel point output after the group of images are subjected to smoothing processing through median filtering is the median of the pixel values in the window with the size of the filtering kernel.
Step S42, overlapping the frames of images in the second image group together to perform laplacian transformation, selecting the largest image in the response values under the same pixel coordinate as the focal plane corresponding to the pixel, and obtaining the depth corresponding to each pixel according to the depth value of the focal plane. Specifically, laplace transform is performed on the image, a focal plane corresponding to each pixel is found, and the focal distance is measured at the same time. According to the camera shooting principle, when a pixel is located on a focal plane, the pixel is in a state where the sharpness or sharpness value is high, whereas the blur or sharpness value is low. And superposing the group of pictures together, performing Laplace transform, and finding out a focal segment value corresponding to the moment when the sharpness of one pixel is maximum, thereby performing depth estimation. LoG filtering is firstly carried out on each picture, and the pixel position with large gradient change in each picture is extracted, namely the LoG filtering can extract the pixel coordinate corresponding to the focal plane when each picture is shot. For the case that each picture has multiple response values at the same pixel coordinate, the image with the largest response value is selected as the focal plane corresponding to the pixel. Meanwhile, the focal plane has relative depth relation due to space-time continuity when moving. The focal plane corresponding to the previous frame of picture taken will be deeper (shallower) than the focal plane of the next frame. According to the acquired focal plane corresponding to each pixel when shooting and the motion model of the movement of the focal plane, namely the change curve of the image in the focusing process, acquired from the previous steps, the distance, namely the depth, of the camera corresponding to each pixel can be acquired.
And step S43, sequentially acquiring the pixel coordinates of each pixel when the contrast ratio is maximum from front to back, recording the pixel coordinates to form the visual angle depth map, and calling the smart phone inertial measurement module to acquire the pose information corresponding to the acquired depth map.
And obtaining a single visual angle depth map by the focal length and the original image, calling the intelligent mobile phone inertial device module, recording the pose of the intelligent mobile phone when the depth map is obtained, and storing the pose. Specifically, the pixel coordinates of each pixel at which the contrast is maximum are sequentially found and recorded from front to back, a depth map of the view angle is formed, and the depth map can be converted into point cloud data as shown in fig. 6. And meanwhile, calling an inertial device module of the smart phone, and recording and storing the pose information of the smart phone when the depth map is acquired.
In this embodiment, the method for performing oral cavity scanning with a smartphone further includes the following steps:
and step S5, converting the acquired single-view depth maps into point cloud data, and iterating by using the pose change translation and rotation matrix between two frames of images and scale information obtained according to the calibration result as initial values to minimize the alignment error of the two point clouds and splicing to form the oral cavity tooth three-dimensional model. Specifically, the above process is repeated to obtain multi-view images, the images corresponding to multiple groups of views are converted into point cloud data, the point cloud data and the pose transformation matrix are sent into the memory, then the multi-view point clouds are spliced, and the spliced point clouds are meshed to form the complete oral digital model shown in fig. 7.
The images of multiple groups of visual angles are registered by using an improved ICP algorithm, and iteration is performed by using pose change translation and rotation matrixes between two frames and scale information estimated according to a calibration result as initial values during splicing, so that the alignment error of two point clouds is minimum, and the oral cavity tooth three-dimensional model is finally obtained. In this embodiment, step S6 may specifically include the following steps.
And registering according to the overlapped parts in the front and back two frames of images, and superposing the target tooth on the corresponding tooth position. Specifically, point cloud sets a and B of two frames before and after are down-sampled as follows:
wherein A isl，BlRespectively representing point cloud sets A and B, f at a resolution level lsample↓Indicating a down-sampling operation, L is the set maximum resolution level,
representing a single sequence number i in a set A of point clouds at a resolution level lThe point cloud a is obtained by the following steps,
the number of the point clouds representing the point cloud set A is a positive integer, and n points are represented;
represents a single point cloud B with the sequence number j in the point cloud set B at the resolution level l,
the number of the point clouds representing the point cloud set B is a positive integer, and the number of the point clouds is m.
Obtaining two point cloud sets A by using nearest neighbor searchlAnd BlBi-directional transformation relation at resolution l (k-1)thAs follows:
wherein K represents the number of iterations,
respectively expressed in (k-1) iterations at a resolution of l, such that
The values of the variables i and j with the smallest error.
Respectively representing a scaling matrix, a rotation matrix and a translation matrix which are needed to be used in the transformation in (k-1) iterations under the resolution level l.
Represents a single point cloud a with the sequence number i in the point cloud set A under the resolution level l,
a single point cloud B with sequence number j in the point cloud set B at resolution level i is represented.
Calculating new transformation matrix between two point clouds based on the transformation matrix H obtained in the previous step, and updating
And
. Wherein
And
respectively representing a scaling matrix, a rotation matrix and a translation matrix which are needed to be used in k iterations under the resolution level l.
The above two steps are repeated until the error precision reaches the resolution l.
The previous step is repeated until the resolution l is 1.
After the splicing is finished, a three-dimensional model of the whole tooth can be formed. The dentist can conveniently prepare teeth according to the model, and then a series of follow-up diagnosis and treatment schemes can be developed.
The method for scanning the oral cavity by using the smart phone disclosed by the embodiment comprises the steps of performing three-dimensional reconstruction on teeth in the oral cavity by using a lens module of the smart phone as an oral cavity scanning device, acquiring a series of images with different depths of field in the oral cavity of a patient through high-speed photography, deblurring and filtering acquired data, aligning the series of images, using a last image as a calibration reference, matching each deblurred image with the calibration, calculating the zoom ratio between the front image and the back image, zooming the images according to the zoom ratio to eliminate the near-far effect, overlapping the images after performing Laplace transform filtering to form a single-view depth image, finally acquiring depth images of multiple view angles to form a point cloud, registering, splicing, and gridding to form a final three-dimensional model. Therefore, the real image in the internal environment of the oral cavity of the patient is transmitted into the smart phone, the real model of the oral cavity is restored, the cost of the oral cavity scanning business is greatly reduced, and the efficiency is improved. The problem of present oral cavity scanner all have the oral cavity scanner and with the huge host computer station constitution of oral cavity scanner connection is solved, the doctor sight need constantly move between oral cavity and body side host computer station screen when the operation, cause scanning efficiency to reduce, huge host computer station leads to whole oral cavity scanner system to occupy a large amount of limited diagnosis and treatment space and the cost is high in addition.
The invention also discloses an oral cavity scanning device, as shown in fig. 8, comprising an external probe 1 with an optical image transmission member inside and a mobile terminal 2 detachably connected with the external probe, wherein the mobile terminal 2 is provided with a camera module capable of acquiring transmission images of the external probe, an inertial measurement module capable of acquiring the pose state of the mobile terminal and a control module, and the control module specifically comprises: the calibration module is used for controlling the camera module to shoot the calibration plate at different distances and calibrating the moving distance of the focal plane in the focusing process; the focal length range acquisition module is used for acquiring a first focal length of the camera module when the camera module forms the clearest image on the surface of a tooth, and zooming the first focal length to acquire a focal length range capable of covering the surface of the tooth; the first image group module is used for controlling the camera module to take a picture at a high speed and simultaneously zoom quickly in the focal range to obtain a group of continuous initial images in the oral cavity, and capturing different depth-of-field images in the group of continuous initial images as a first image group according to the focal range; and the single-view depth map generation module is used for aligning each frame of image in the first image group, then taking the last image as a reference, sequentially matching the last image with the previous image to calculate a zoom factor, zooming each frame of image in the first image group according to the zoom factor, then overlapping to form a single-view depth map, and calling the intelligent mobile phone inertia measurement module to acquire corresponding pose information when the group of depth maps is acquired.
In this embodiment, the control module of the mobile terminal further includes a three-dimensional model generation module, configured to convert the acquired single-view depth maps into point cloud data, and iterate using a pose change translation and a rotation matrix between two frames of images and scale information obtained according to a calibration result as an initial value, so that an alignment error of the two point clouds is minimized, and an oral cavity tooth three-dimensional model is formed by stitching.
In this embodiment, the single-view depth map generating module specifically includes: the second image group acquisition module is used for sequentially selecting two frames of images before and after the first image group, searching the images by using an SIFT algorithm to obtain respective feature points, solving the transformation relation of the two groups of feature points by using an RANSAC algorithm, solving a corresponding homography matrix, and transforming the second image by using the solved matrix to obtain a transformed second image group consisting of the images with the near-far effect eliminated; the depth acquisition module is used for superposing the frames of images in the second image group together to perform Laplace transform, selecting the largest image in response values under the same pixel coordinate as a focal plane corresponding to the pixel, and acquiring the depth corresponding to the pixel according to the depth value of the focal plane; and the depth map module is used for sequentially acquiring the pixel coordinates of each pixel when the contrast is maximum from front to back, recording the pixel coordinates to form the visual angle depth map, and calling the smart phone inertial measurement module to acquire the pose information corresponding to the acquired depth map.
In this embodiment, the second image group acquiring module specifically includes: the feature point module is used for searching front and rear frames of images in the first image group by using an SIFT algorithm to obtain respective feature points; the homography matrix acquisition module is used for randomly selecting a plurality of groups of matched characteristic points to calculate a homography matrix, calculating errors of other matched characteristic point pairs after homography matrix transformation, acquiring the number of the characteristic point pairs with the errors smaller than a threshold value, and acquiring the homography matrix with the largest number of the characteristic point pairs with the errors smaller than the threshold value; the transformed image acquisition module is used for transforming the second image by using the solved homography matrix to obtain a transformed image for eliminating the near-far effect; and the second image generation module is used for continuously transforming the front frame image and the rear frame image in the first image group to obtain a transformed image for eliminating the near-far effect and form a second image group.
The specific components and functions of the control module of the oral cavity scanning device correspond to those of the method for oral cavity scanning with a smartphone disclosed in the foregoing embodiments one to one, so that detailed descriptions are not provided herein, and reference may be made to each of the embodiments of the method for oral cavity scanning with a smartphone disclosed above. It should be noted that, in the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other.
In further embodiments, there is also provided an apparatus for oral scanning with a smartphone, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, when executing the computer program, implementing the steps of the method for oral scanning with a smartphone as described in the embodiments above.
The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general processor may be a microprocessor or the processor may be any conventional processor, etc., and the processor is a control center of the apparatus for oral cavity scanning with the smart phone, and various interfaces and lines are used to connect various parts of the whole apparatus for oral cavity scanning with the smart phone.
The memory can be used for storing the computer program and/or module, and the processor can realize various functions of the device and the equipment for oral cavity scanning by the smart phone by running or executing the computer program and/or module stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like, and the memory may include a high speed random access memory, and may further include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
The device for oral cavity scanning with a smart phone can be stored in a computer readable storage medium if it is implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, all or part of the processes of the method of the embodiments described above can be implemented by a computer program, which can be stored in a computer readable storage medium and can be executed by a processor to implement the steps of the embodiments of the method for performing oral cavity scanning with a smartphone. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
In summary, the above-mentioned embodiments are only preferred embodiments of the present invention, and all equivalent changes and modifications made in the claims of the present invention should be covered by the claims of the present invention.
Claims (10)
1. A method for scanning oral cavity by using a smart phone, wherein the front end of a camera lens of the smart phone is connected with an external probe which can be used for transmitting an image of teeth in the oral cavity, the method is characterized by comprising the following steps:
s1, calibrating a camera module of the smart phone, controlling the camera module to shoot the calibration plate at different distances, and calibrating the moving distance of a focal plane in the focusing process;
s2, acquiring a first focal length of the camera module when the camera module forms the clearest image on the surface of a tooth, and zooming the first focal length to obtain a focal length range capable of covering the surface of the tooth;
s3, controlling the camera module to take a picture at a high speed and zoom quickly in the focal range to obtain a group of continuous initial images in the oral cavity, and capturing different depth-of-field images in the group of continuous initial images as a first image group according to the focal range;
and S4, aligning each frame image in the first image group, sequentially matching the frame images with the previous image by taking the last image as a reference to calculate a zoom factor, zooming each frame image in the first image group according to the zoom factor, superposing to form a single-view depth map, and calling an intelligent mobile phone inertia measurement module to acquire corresponding pose information when the group of depth maps is acquired.
2. The method of claim 1, further comprising the steps of:
and S5, converting the acquired single-view depth maps into point cloud data, and iterating by using the pose change translation and rotation matrix between the two frames of images and scale information obtained according to the calibration result as initial values to minimize the alignment error of the two point clouds and splicing to form the oral cavity tooth three-dimensional model.
3. The method for oral cavity scanning with a smartphone according to claim 2, wherein the step S4 specifically includes:
s41, sequentially selecting front and rear frames of images of the first image group, searching by using an SIFT algorithm to obtain respective feature points, solving the transformation relation of the two sets of feature points by using a RANSAC algorithm, solving a corresponding homography matrix, and transforming the second image by using the solved matrix to obtain a transformed second image group consisting of images with near-far small effect eliminated;
s42, overlapping the images of each frame in the second image group together for Laplace transform, selecting the largest image in the response values under the same pixel coordinate as the focal plane corresponding to the pixel, and obtaining the depth corresponding to each pixel according to the depth value of the focal plane;
and S43, sequentially acquiring the pixel coordinates of each pixel when the contrast ratio is maximum from front to back, recording the pixel coordinates to form the visual angle depth map, and calling the smart phone inertial measurement module to acquire the pose information corresponding to the acquired depth map.
4. The method for oral cavity scanning with a smartphone according to claim 3, wherein the step S41 specifically includes:
s411, searching front and back frames of images in the first image group by using an SIFT algorithm to obtain respective feature points;
s412, randomly selecting a plurality of groups of matched feature points to calculate a homography matrix, calculating errors of other matched feature points after homography matrix transformation, obtaining the number of the feature points with the errors smaller than a threshold value, and obtaining the homography matrix with the largest number of the feature points with the errors smaller than the threshold value;
s413, transforming the second image by using the solved homography matrix to obtain a transformed image for eliminating the near-far effect;
and S414, continuously adopting the steps to sequentially transform the front frame image and the back frame image in the first image group to obtain a transformed image without the near-far effect, and forming a second image group.
5. The utility model provides an oral cavity scanning device, including inside have optical image transfer member external probe and with external probe separable connection's mobile terminal, mobile terminal has the module of making a video recording that can acquire external probe transmission image, can acquire inertial measurement module and the control module group of mobile terminal position appearance state, the control module group specifically includes:
the calibration module is used for controlling the camera module to shoot the calibration plate at different distances and calibrating the moving distance of the focal plane in the focusing process;
the focal length range acquisition module is used for acquiring a first focal length of the camera module when the camera module forms the clearest image on the surface of a tooth, and zooming the first focal length to acquire a focal length range capable of covering the surface of the tooth;
the first image group module is used for controlling the camera module to take a picture at a high speed and simultaneously zoom quickly in the focal range to obtain a group of continuous initial images in the oral cavity, and capturing different depth-of-field images in the group of continuous initial images as a first image group according to the focal range;
and the single-view depth map generation module is used for aligning each frame of image in the first image group, then taking the last image as a reference, sequentially matching the last image with the previous image to calculate a zoom factor, zooming each frame of image in the first image group according to the zoom factor, then overlapping to form a single-view depth map, and calling the intelligent mobile phone inertia measurement module to acquire corresponding pose information when the group of depth maps is acquired.
6. The oral scanning device of claim 5, wherein the control module further comprises:
and the three-dimensional model generation module is used for converting the acquired single-view depth maps into point cloud data, and iterating by using the pose change translation and rotation matrix between the two frames of images and scale information obtained according to the calibration result as an initial value to minimize the alignment error of the two point clouds and form the oral cavity tooth three-dimensional model by splicing.
7. The oral scanning device of claim 6, wherein the single-view depth map generation module specifically comprises:
the second image group acquisition module is used for sequentially selecting two frames of images before and after the first image group, searching the images by using an SIFT algorithm to obtain respective feature points, solving the transformation relation of the two groups of feature points by using an RANSAC algorithm, solving a corresponding homography matrix, and transforming the second image by using the solved matrix to obtain a transformed second image group consisting of the images with the near-far effect eliminated;
the depth acquisition module is used for superposing the frames of images in the second image group together to perform Laplace transform, selecting the largest image in response values under the same pixel coordinate as a focal plane corresponding to the pixel, and acquiring the depth corresponding to the pixel according to the depth value of the focal plane;
and the depth map module is used for sequentially acquiring the pixel coordinates of each pixel when the contrast is maximum from front to back, recording the pixel coordinates to form the visual angle depth map, and calling the smart phone inertial measurement module to acquire the pose information corresponding to the acquired depth map.
8. The oral cavity scanning device according to claim 7, wherein the second image group acquiring module specifically comprises:
the feature point module is used for searching front and rear frames of images in the first image group by using an SIFT algorithm to obtain respective feature points;
the homography matrix acquisition module is used for randomly selecting a plurality of groups of matched characteristic points to calculate a homography matrix, calculating errors of other matched characteristic point pairs after homography matrix transformation, acquiring the number of the characteristic point pairs with the errors smaller than a threshold value, and acquiring the homography matrix with the largest number of the characteristic point pairs with the errors smaller than the threshold value;
the transformed image acquisition module is used for transforming the second image by using the solved homography matrix to obtain a transformed image for eliminating the near-far effect;
and the second image generation module is used for continuously transforming the front frame image and the rear frame image in the first image group to obtain a transformed image for eliminating the near-far effect and form a second image group.
9. An apparatus for oral scanning with a smartphone, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: the processor, when executing the computer program, realizes the steps of the method according to any of claims 1-4.
10. A computer-readable storage medium storing a computer program, characterized in that: the computer program realizing the steps of the method according to any of claims 1-4 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111545540.4A CN114283236A (en) | 2021-12-16 | 2021-12-16 | Method, device and storage medium for oral cavity scanning by using smart phone |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111545540.4A CN114283236A (en) | 2021-12-16 | 2021-12-16 | Method, device and storage medium for oral cavity scanning by using smart phone |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114283236A true CN114283236A (en) | 2022-04-05 |
Family
ID=80872692
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111545540.4A Pending CN114283236A (en) | 2021-12-16 | 2021-12-16 | Method, device and storage medium for oral cavity scanning by using smart phone |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114283236A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114886345A (en) * | 2022-04-20 | 2022-08-12 | 青岛海尔空调器有限总公司 | Method, device and system for controlling sweeping robot and storage medium |
-
2021
- 2021-12-16 CN CN202111545540.4A patent/CN114283236A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114886345A (en) * | 2022-04-20 | 2022-08-12 | 青岛海尔空调器有限总公司 | Method, device and system for controlling sweeping robot and storage medium |
| CN114886345B (en) * | 2022-04-20 | 2023-12-15 | 青岛海尔空调器有限总公司 | Method, device, system and storage medium for controlling sweeping robot |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | Zoom to learn, learn to zoom | |
| Wronski et al. | Handheld multi-frame super-resolution | |
| Wadhwa et al. | Synthetic depth-of-field with a single-camera mobile phone | |
| JP6347675B2 (en) | Image processing apparatus, imaging apparatus, image processing method, imaging method, and program | |
| Green et al. | Multi-aperture photography | |
| JP3961729B2 (en) | All-focus imaging device | |
| US9749601B2 (en) | Imaging device, image display method, and storage medium for displaying reconstruction image | |
| CN108055452A (en) | Image processing method, device and equipment | |
| CN110689581A (en) | Structured light module calibration method, electronic device and computer readable storage medium | |
| CN110536057A (en) | Image processing method and device, electronic equipment, computer readable storage medium | |
| JP2009054130A (en) | Image processor, image processing method, and digital still camera | |
| CN106034203B (en) | Image processing method and device for shooting terminal | |
| US10943333B2 (en) | Method and apparatus of sharpening of gastrointestinal images based on depth information | |
| CN104939793A (en) | Variable-focus 3-D capsule endoscope system based on liquid lens | |
| JP7378219B2 (en) | Imaging device, image processing device, control method, and program | |
| JP2013026844A (en) | Image generation method and device, program, recording medium, and electronic camera | |
| WO2014002521A1 (en) | Image processing device and image processing method | |
| WO2018032841A1 (en) | Method, device and system for drawing three-dimensional image | |
| CN117061868A (en) | Automatic photographing device based on image recognition | |
| CN114283236A (en) | Method, device and storage medium for oral cavity scanning by using smart phone | |
| WO2020134123A1 (en) | Panoramic photographing method and device, camera and mobile terminal | |
| CN112261399B (en) | Capsule endoscope image three-dimensional reconstruction method, electronic device and readable storage medium | |
| JP6624785B2 (en) | Image processing method, image processing device, imaging device, program, and storage medium | |
| JP7489253B2 (en) | Depth map generating device and program thereof, and depth map generating system | |
| CN117058183A (en) | Image processing method and device based on double cameras, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |