CN117765042A

CN117765042A - Registration method and device for oral tomographic image, computer equipment and storage medium

Info

Publication number: CN117765042A
Application number: CN202311809536.3A
Authority: CN
Inventors: 陈云; 盛鸿; 章征贵; 刘技; 陶艳
Original assignee: Suzhou Dikaier Medical Technology Co ltd
Current assignee: Suzhou Dikaier Medical Technology Co ltd
Priority date: 2023-12-26
Filing date: 2023-12-26
Publication date: 2024-03-26

Abstract

The present disclosure relates to a method, apparatus, computer device, storage medium for registration of oral tomographic images. The method comprises the following steps: acquiring an oral cavity tomographic image and a binocular image and determining a first registration relationship between a tomographic coordinate system corresponding to the oral cavity tomographic image and a binocular camera coordinate system corresponding to the binocular image; based on the first registration relation and parameters of the binocular camera, projecting the three-dimensional data of the oral cavity in the fault coordinate system onto the binocular image to generate a fault projection image; determining a second registration relation and a projection error between a tomographic coordinate system and a binocular camera coordinate system according to the tomographic projection image and the characteristic points in the oral tomographic image; and determining a conversion relation between a reference coordinate system corresponding to the reference device and a fault coordinate system based on the second registration relation and the pose of the reference device in the binocular image in response to the projection error being smaller than a preset threshold value, and registering based on the conversion relation. The method can reduce the registering difficulty and improve the registering accuracy.

Description

Registration method and device for oral tomographic image, computer equipment and storage medium

Technical Field

The disclosure relates to the technical field of spatial calibration, in particular to a registration method, a registration device, computer equipment and a storage medium of an oral tomographic image.

Background

With the development of medical instruments, for operations such as dental implant and the like, accurate positioning and guiding can be performed by means of an optical navigation system in order to ensure the planting precision. The optical navigation system is generally composed of a binocular camera, a computer, a display and the like, can realize accurate positioning of the instrument containing the specified characteristics in the scene, plans a movement route according to the difference between the current pose and the target pose, displays the movement route in the display and guides the movement of the instrument in real time.

The optical navigation planting flow is as follows: shooting the oral cavity CBCT (Cone Beam Computed Tomography) of a patient before operation, planning the implantation position, solidifying a reference device with specified characteristics on the dental jaw to be implanted of the patient by optical navigation implantation, and establishing a conversion relation between a coordinate system of the reference device and a data coordinate system of the dental jaw CT (Computed Tomography) through a registration process; during operation, the position and posture difference between the mobile phone and the target implant tooth position is obtained through the positioning reference device of the optical navigation system and the surgical instrument (such as the mobile phone) fixedly connected with the designated characteristics, and the three-dimensional visualization software is combined, so that an implant path can be planned, and a doctor is guided to complete high-precision implantation in real time.

However, a conventional registration procedure requires an additional CBCT, which increases the radiation dose of the patient and is relatively complex. The other dental cusp registration process requires a doctor to manually select characteristic points or surfaces, has higher requirements on the operation experience of the doctor, is more complicated to operate, and can generate larger errors if the characteristic points or surfaces are selected incorrectly.

Disclosure of Invention

Based on the above, it is necessary to provide a method, an apparatus, a computer device, and a storage medium for registering an oral tomographic image, which reduce the difficulty of registration and improve the accuracy of registration.

In a first aspect, the present disclosure provides a method of registration of an oral tomographic image. The method comprises the following steps:

acquiring an oral cavity tomographic image of a target object and a binocular image in an oral cavity after a reference device is arranged in the oral cavity of the target object, wherein the oral cavity tomographic image is obtained after computed tomography, and the binocular image is obtained after shooting by a binocular camera;

determining a first registration relationship between a tomographic coordinate system corresponding to the oral tomographic image and a binocular camera coordinate system corresponding to the binocular image according to the features in the oral tomographic image and the binocular image;

Based on the first registration relation and parameters of the binocular camera, projecting the three-dimensional data of the oral cavity in the fault coordinate system onto the binocular image to generate a fault projection image;

determining a second registration relationship and a projection error between the tomographic coordinate system and the binocular camera coordinate system according to the tomographic projection image and the feature points in the oral tomographic image;

and determining a conversion relation between a reference coordinate system corresponding to the reference device and the fault coordinate system based on the second registration relation and the pose of the reference device in the binocular image in response to the projection error being smaller than a preset threshold, and registering based on the conversion relation.

In one embodiment, the determining, according to the features in the oral tomographic image and the binocular image, a first registration relationship between a tomographic coordinate system corresponding to the oral tomographic image and a binocular camera coordinate system corresponding to the binocular image includes:

extracting a first feature corresponding to a target position in the oral cavity from the oral cavity tomogram, and determining a first coordinate of the first feature in a tomo coordinate system corresponding to the oral cavity tomogram;

Extracting a second feature corresponding to the target position from the binocular image, and determining a second coordinate of the second feature in a binocular camera coordinate system corresponding to the binocular image;

a first registration relationship between the tomographic coordinate system and the binocular camera coordinate system is determined based on the first and second coordinates.

In one embodiment, the determining a first registration relationship between the tomographic coordinate system and the binocular camera coordinate system based on the first and second coordinates includes:

pairing the first coordinate and the second coordinate by using a pairing search algorithm;

and processing the pairing result by using an iterative nearest neighbor optimization algorithm, and determining a first registration relationship between the fault coordinate system and the binocular camera coordinate system.

In one embodiment, the determining the second registration relationship between the tomographic coordinate system and the binocular camera coordinate system and the projection error according to the tomographic projection image and the feature points in the oral tomographic image includes:

extracting projection two-dimensional features and camera two-dimensional features from the tomographic projection image; the two-dimensional projection features are features of the target position in the three-dimensional data of the oral cavity projected onto the binocular image, and the two-dimensional camera features are features of the target position on the binocular image;

Determining a second registration relationship between a tomographic coordinate system and the binocular camera coordinate system based on the projected two-dimensional features and the camera two-dimensional features;

and calculating to obtain the projection error based on the tomographic projection image, characteristic points in the oral tomographic image, characteristic coordinates corresponding to the characteristic points, parameters of the binocular camera and the second registration relationship.

In one embodiment, the extracting the projection two-dimensional feature and the camera two-dimensional feature in the tomographic projection image includes:

determining a target position at an image projected by the oral cavity three-dimensional data in the tomographic projection image, and determining a projected two-dimensional feature based on a normal corresponding to the target position;

and determining a two-dimensional camera characteristic based on the target position in the binocular image in the tomographic projection image.

In one embodiment, the calculating the projection error based on the feature points in the tomographic image and the oral tomographic image, feature coordinates corresponding to the feature points, parameters of the binocular camera, and the second registration relationship includes:

and calculating to obtain a projection error according to the number of feature points in the two-dimensional features of the camera in the left-eye image and the right-eye image of the binocular image in the tomographic projection image, the coordinates of the feature points, the coordinates of three-dimensional feature points corresponding to the feature points in the two-dimensional features of the camera in the oral tomographic image, the parameters of the binocular camera and the second registration relation.

In one embodiment, the method further comprises:

adjusting the second registration relationship in response to the projection error being greater than or equal to the preset threshold;

and projecting the three-dimensional data of the oral cavity in the fault coordinate system onto the binocular image by using the adjusted second registration relation and the parameters of the binocular camera, regenerating a fault projection image and calculating the projection error until the projection error is smaller than or equal to the preset threshold value.

In a second aspect, the present disclosure also provides a registration apparatus for oral tomographic images. The device comprises:

the image acquisition module is used for acquiring an oral cavity tomogram of a target object and a binocular image in the oral cavity after the reference device is arranged in the oral cavity of the target object, wherein the oral cavity tomogram is obtained after computed tomography, and the binocular image is obtained after shooting by a binocular camera;

a first relation determining module, configured to determine a first registration relation between a tomographic coordinate system corresponding to the oral tomographic image and a binocular camera coordinate system corresponding to the binocular image according to features in the oral tomographic image and the binocular image;

The image generation module is used for projecting the oral cavity three-dimensional data in the fault coordinate system onto the binocular image based on the first registration relation and the parameters of the binocular camera to generate a fault projection image;

the computing module is used for determining a second registration relation and a projection error between the fault coordinate system and the binocular camera coordinate system according to the fault projection image and the characteristic points in the oral cavity fault image;

and the registration module is used for responding to the projection error being smaller than a preset threshold value, determining a conversion relation between a reference coordinate system corresponding to the reference device and the fault coordinate system based on the second registration relation and the pose of the reference device in the binocular image, and registering based on the conversion relation.

In a third aspect, the present disclosure also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of any of the method embodiments described above when the processor executes the computer program.

In a fourth aspect, the present disclosure also provides a computer-readable storage medium. The computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of any of the method embodiments described above.

In a fifth aspect, the present disclosure also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of any of the method embodiments described above.

In the above embodiments, by acquiring the oral tomographic image and the binocular image, the first registration relationship between the tomographic coordinate system and the binocular camera coordinate system can be determined based on the features in the images. The patient only needs to take a CT once. Because of the different dimensions that the tomographic coordinate system and the binocular camera coordinate system belong to, the first registration relationship cannot generally be regarded as the most accurate registration relationship, but can only be regarded as the coarse registration relationship. At the moment, the three-dimensional data of the oral cavity in the fault coordinate system can be projected onto the binocular image according to the first registration relation and the parameters of the binocular camera to generate a fault projection image. And then the three-dimensional data projection is changed into two-dimensional data. And determining a second registration relation and a projection error between the tomographic coordinate system and the binocular camera coordinate system by using the tomographic projection image and the characteristic points in the oral tomographic image. An accurate second registration relationship between the tomographic coordinate system and the binocular camera coordinate system in two dimensions can be determined. In order to verify whether the second registration relationship meets the requirement, a projection error can be calculated, and when the projection error is smaller than a preset threshold value, a conversion relationship between a reference coordinate system corresponding to the reference device and a fault coordinate system can be determined based on the second registration relationship and the pose of the reference device in the binocular image, and registration is performed based on the conversion relationship. Only the oral cavity tomographic image and the binocular image are needed to be obtained, so that a doctor does not need to have higher operation experience, the characteristic points do not need to be accurately selected, and the complexity is reduced. In addition, the two registration relations, namely the first registration relation and the second registration relation, are used, and the projection errors are utilized for verification, so that the accuracy of registration can be improved.

Drawings

In order to more clearly illustrate the embodiments of the present disclosure or the prior art, the drawings that are required in the detailed description or the prior art will be briefly described, it will be apparent that the drawings in the following description are some embodiments of the present disclosure, and other drawings may be obtained according to the drawings without inventive effort for a person of ordinary skill in the art.

FIG. 1 is a schematic view of an application environment of a method for registering an oral tomographic image in one embodiment;

FIG. 2 is a flow chart of a method of registration of an oral tomographic image in one embodiment;

FIG. 3 is a flow chart of step S204 in one embodiment;

FIG. 4 is a flow chart of step S306 in one embodiment;

FIG. 5 is a flow chart of step S208 in one embodiment;

FIG. 6 is a flow chart of step S502 in one embodiment;

FIG. 7 is a flow chart of a method of registration of an oral tomographic image in another embodiment;

FIG. 8 is a block diagram schematically illustrating a structure of a registration apparatus for oral tomographic images in one embodiment;

FIG. 9 is a schematic diagram of the internal structure of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present disclosure more apparent, the present disclosure will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present disclosure.

It should be noted that the terms "first," "second," and the like in the description and claims herein and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or device that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or device.

In this document, the term "and/or" is merely one association relationship describing the associated object, meaning that three relationships may exist. For example, a and/or B may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

As described in the background, current registration schemes generally include: CT visualization registration, for which the reference device is already fixed to the dental jaw. By fixing the development feature on the reference device (typically, the relative position of the development feature and the reference device is known from the factory). Then, the patient wears the reference device to shoot CT, three-dimensional coordinates of the dental jaw and the developing characteristic are extracted from the CT, and a coordinate system conversion relation between the dental jaw CT and the CT developing characteristic and the reference device is established, so that registration is realized. Because the patient needs to shoot CBCT once before operation, and still needs to shoot CT once again after wearing the reference device, the radiation dose received by the patient is increased, and the shooting results of the two CT are needed to be integrated, so that the configuration flow is more complicated.

Current registration schemes also typically include: in the cusp registration mode, after a patient wears a reference device, a plurality of characteristic points or surfaces are automatically recommended or selected on a jaw CT (computed tomography), then a doctor controls a cusp selection instrument (the cusp point of which can be tracked by an optical navigation system), corresponding characteristic points or surfaces are selected on a jaw entity of the patient, the optical navigation system simultaneously positions and obtains the positions of the reference instrument and the entity characteristic points or surfaces, and the coordinate system conversion relation between the reference instrument, the entity characteristic points, the CT characteristic points and the jaw CT is established by combining with the characteristic point information in the CT, so that registration is realized. However, for this way, since the doctor is required to operate the cusp selecting apparatus, the corresponding feature points on the dental entity need to be accurately selected according to the recommended points in the CT, and the requirement on the operation experience of the doctor is high, otherwise, a large error is generated. In addition, to ensure the selection accuracy, more points or planes are generally selected, increasing the complexity of the operation.

In addition, the traditional technology also has a mode of registration without using a reference device, and the mode obtains the space geometrical relationship between the binocular camera coordinate system of the optical navigation system and the dental CT coordinate system by means of the three-dimensional scanning data of the oral cavity and the tooth information which can be observed all the time, so as to perform configuration. However, this method also requires additional acquisition of three-dimensional data of the oral cavity of the patient, and also requires always observable tooth information to be configured, which is limited in application scenario.

Accordingly, to solve the above-mentioned problems, an embodiment of the present disclosure provides a method for registering an oral tomographic image, which can be applied to an application environment as shown in fig. 1. Wherein the terminal 102 communicates with the binocular camera 104, display, etc. The terminal 102 is provided with a storage unit for storing various images, data, and the like. The terminal 102 may acquire an oral tomographic image of the target object stored in the storage unit. The binocular camera 104 may also capture binocular images of the target subject after the reference device is installed in the mouth of the subject, resulting in binocular images. The terminal 102 may also acquire the binocular image at this time. The terminal 102 may determine a first registration relationship between a tomographic coordinate system corresponding to the oral tomographic image and a binocular camera coordinate system corresponding to the binocular image according to features in the oral tomographic image and the binocular image. The terminal 102 may further project the oral three-dimensional data in the tomographic coordinate system onto the binocular image based on the first registration relationship and parameters of the binocular camera 104, generating a tomographic projection image. The terminal 102 may determine a second registration relationship between the tomographic coordinate system and the binocular camera coordinate system and a projection error from the tomographic projection image, the feature points in the oral tomographic image. In response to the projection error being smaller than a preset threshold, the terminal 102 may determine a conversion relationship between a reference coordinate system corresponding to the reference device and the fault coordinate system based on the second registration relationship and the pose of the reference device in the binocular image, and perform registration based on the conversion relationship. The terminal 102 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, etc.

In one embodiment, as shown in fig. 2, a method for registering an oral tomographic image is provided, which is illustrated by taking the application of the method to the terminal 102 in fig. 1 as an example, and includes the following steps:

s202, acquiring an oral cavity tomographic image of a target object and a binocular image in an oral cavity after a reference device is arranged in the oral cavity of the target object, wherein the oral cavity tomographic image is obtained after computed tomography, and the binocular image is obtained after shooting by a binocular camera.

Wherein the target object may generally be a patient in need of oral surgery in some embodiments of the present disclosure. The oral tomographic image may be an oral CT image or an oral CBCT image in the oral cavity in some embodiments of the present disclosure. The oral cavity CBCT (Cone Beam Computed Tomography) is a medical imaging technique that generates high resolution images of three-dimensional oral structures using cone beam X-rays and a computer reconstruction algorithm. Compared to conventional oral X-rays, oral CBCT can provide more detailed, accurate oral structural information including teeth, maxilla, mandible, temporomandibular joint, etc. The reference means may typically be a reference plate. The reference plate is a precisely positioned device that, by being fixed in the patient's mouth, can provide accurate spatial positioning information and reference points for anatomical structures to assist the surgeon in performing precise procedures during oral surgery. In an oral implant procedure, the reference plate may be used to determine the position and angle of the implant, thereby ensuring accurate implantation of the implant. Through the reference plate, doctors can perform operations under the navigation of the computer, accurately determine the position of the implant, avoid damaging peripheral nerves and blood vessels, and improve the accuracy and safety of the operations. In addition, typically, the surface of the reference device is typically provided with designated features that can be observed and located by the optical navigation system.

The computed tomography scan may be a CT scan. A binocular camera is a camera system with two cameras that simulates the visual system of both eyes of a human being. Such a camera system may be used to obtain three-dimensional depth information.

Specifically, when the target object needs to perform an oral surgery, firstly, an oral tomographic image may be photographed, and then the oral tomographic image of the target object may be transmitted to the terminal. The reference device may then be secured in the mouth of the target subject. Typically, the reference device will be fixed to the jaw of the target subject. After the reference device is fixed, the binocular camera can be used for shooting the oral cavity of the target object to obtain binocular images. It should be noted that, since the present disclosure requires registration, the binocular image typically includes the reference device and the teeth.

In some exemplary embodiments, to ensure that the reference device and teeth are included in the binocular image, the patient's teeth may be maintained substantially parallel to the binocular camera while taking a photograph with the binocular camera.

S204, determining a first registration relation between a fault coordinate system corresponding to the oral cavity fault image and a binocular camera coordinate system corresponding to the binocular image according to the characteristics in the oral cavity fault image and the binocular image.

Wherein the feature may generally be a location in the mouth having a particular shape, indicia. Features may include: occlusal surface boundaries, shape of a tooth, boundaries, etc. The tomographic coordinate system may be a CT coordinate system, that is, a coordinate system corresponding to the oral tomographic image. The binocular camera coordinate system may generally be the coordinate system to which the binocular camera corresponds. The first registration relationship may generally include a transformation matrix, an amount of translation, and the like between the tomographic coordinate system and the binocular camera coordinate system.

In particular, a first registration relationship between the tomographic coordinate system and the binocular camera coordinate system may be determined from features of the same location in the oral tomographic image and the binocular image.

S206, based on the first registration relation and the parameters of the binocular camera, projecting the oral three-dimensional data in the fault coordinate system onto the binocular image to generate a fault projection image.

Wherein, the tomographic projection image may generally include: the original binocular image and the two-dimensional oral image after the oral three-dimensional data are projected to the binocular image. Parameters of the binocular camera may include: an internal reference of the binocular camera and an external reference of the binocular camera. The external parameters of the binocular camera include the relative position and direction of the cameras, i.e., the relative position and rotation angle between the left and right cameras, etc. The internal parameters include internal parameters of the camera, such as focal length, principal point position, distortion, etc., for correcting the image. Specifically, the internal parameters of the binocular camera include an internal parameter matrix, distortion coefficients, rotation matrix, and translation vector of the left and right cameras. The oral three-dimensional data may generally be data in an oral tomographic image.

Specifically, according to the first registration relationship and in combination with parameters of the binocular camera, coordinate conversion is performed on three-dimensional data of the oral cavity under the fault coordinate system, and then the three-dimensional data are projected onto the binocular image to generate a fault projection image.

And S208, determining a second registration relation and a projection error between the tomosynthesis coordinate system and the binocular camera coordinate system according to the tomosynthesis image and the characteristic points in the oral tomosynthesis image.

The projection error may be an error of performing coordinate transformation on the three-dimensional data of the oral cavity and then projecting the three-dimensional data onto the binocular image. The feature points may generally be locations in the oral cavity that have distinct features.

Specifically, since the oral tomographic image is typically three-dimensional data, the binocular image is two-dimensional data. Thus, the first registration relationship may typically be a difference relationship between the three-dimensional coordinate system and the two-dimensional coordinate system. This relationship is typically not very accurate due to the dimensional gap, and can only be used as a coarse registration relationship. In order to further improve the accuracy of registration, since the three-dimensional data of the oral cavity is projected onto the binocular image, the three-dimensional data of the oral cavity is converted into two-dimensional data at this time, and the difference relation between the tomographic coordinate system and the binocular coordinate system can be determined more accurately. The data in the tomographic image may be used to determine a second registration relationship between the tomographic coordinate system and the binocular camera coordinate system, and the projection error between the two and three dimensions is determined from the tomographic data, the oral tomographic image, and the second registration relationship.

And S210, determining a conversion relation between a reference coordinate system corresponding to the reference device and the fault coordinate system based on the second registration relation and the pose of the reference device in the binocular image in response to the projection error being smaller than a preset threshold, and registering based on the conversion relation.

Specifically, when the projection error is smaller than a preset threshold, it may be generally determined that the second registration relationship satisfies the requirement, and at this time, since the second registration relationship is a relationship between the tomographic coordinate system and the binocular camera coordinate system, the pose of the reference device in the binocular image may be further utilized to directly establish a conversion relationship between the reference coordinate system and the tomographic coordinate system, thereby completing the registration process. In the subsequent operation process, the reference device can be positioned through the optical navigation system, the pose of the fault coordinate system where the tooth data are located can be obtained, and the pose of the implant to be implanted can be obtained in real time.

In the method for registering the oral cavity tomogram, the first registration relation between the tomogram and the binocular camera coordinate system can be determined according to the characteristics in the images by acquiring the oral cavity tomogram and the binocular image. The patient only needs to take a CT once. Because of the different dimensions that the tomographic coordinate system and the binocular camera coordinate system belong to, the first registration relationship cannot generally be regarded as the most accurate registration relationship, but can only be regarded as the coarse registration relationship. At the moment, the three-dimensional data of the oral cavity in the fault coordinate system can be projected onto the binocular image according to the first registration relation and the parameters of the binocular camera to generate a fault projection image. And then the three-dimensional data projection is changed into two-dimensional data. And determining a second registration relation and a projection error between the tomographic coordinate system and the binocular camera coordinate system by using the tomographic projection image and the characteristic points in the oral tomographic image. An accurate second registration relationship between the tomographic coordinate system and the binocular camera coordinate system in two dimensions can be determined. In order to verify whether the second registration relationship meets the requirement, a projection error can be calculated, and when the projection error is smaller than a preset threshold value, a conversion relationship between a reference coordinate system corresponding to the reference device and a fault coordinate system can be determined based on the second registration relationship and the pose of the reference device in the binocular image, and registration is performed based on the conversion relationship. Only the oral cavity tomographic image and the binocular image are needed to be obtained, so that a doctor does not need to have higher operation experience, the characteristic points do not need to be accurately selected, and the complexity is reduced. In addition, the two registration relations, namely the first registration relation and the second registration relation, are used, and the projection errors are utilized for verification, so that the accuracy of registration can be improved.

In one embodiment, as shown in fig. 3, the determining, according to the features in the oral tomographic image and the binocular image, a first registration relationship between a tomographic coordinate system corresponding to the oral tomographic image and a binocular camera coordinate system corresponding to the binocular image includes:

s302, extracting a first feature corresponding to a target position in the oral cavity from the oral cavity tomogram, and determining a first coordinate of the first feature in a tomogram corresponding to the oral cavity tomogram.

Wherein the target location may generally be a location having some overt features in some embodiments of the present disclosure, which facilitates localization.

Specifically, since the oral tomographic image typically has a plurality of angles, the first registration relationship obtained is accurate. An oral tomographic image having the same angle as the binocular image, i.e., the oral tomographic image has the same angle as the binocular image, can be found. For example, the oral tomographic image and the binocular image may both be orthographic projection images of the anterior teeth of the oral cavity. A target location in the mouth, such as a occlusal boundary of a tooth, may be determined, which is in turn determined in the mouth tomographic image, and a first feature corresponding to the target location is extracted. And determining a first coordinate of the first feature in a fault coordinate system.

S304, extracting a second feature corresponding to the target position from the binocular image, and determining a second coordinate of the second feature in a binocular camera coordinate system corresponding to the binocular image.

Specifically, since the oral tomographic image is three-dimensional data, in order to further improve accuracy of the first registration relationship, a target position in the binocular image may be extracted by an image processing method first, so as to obtain a first feature corresponding to the target position, and then the target position and the feature in the binocular image are matched and three-dimensionally reconstructed by using principles such as epipolar constraint and template matching, so as to obtain coordinates of the first feature under a binocular camera coordinate system.

S306, determining a first registration relation between the fault coordinate system and the binocular camera coordinate system based on the first coordinate and the second coordinate.

Specifically, after coordinates of the same position in different coordinate systems are obtained, a first registration relationship between the tomographic coordinate system and the binocular camera coordinate system can be determined using a coordinate difference between the first coordinate and the second coordinate.

In this embodiment, the features of the same target position are determined in the oral tomographic image and the binocular image, and then the coordinates of the features in different coordinate systems are determined, so that the difference (the first registration relationship) between the different coordinate systems can be accurately obtained by using the coordinates, and the accuracy of registration can be ensured.

In one embodiment, as shown in fig. 4, the determining the first registration relationship between the tomographic coordinate system and the binocular camera coordinate system based on the first and second coordinates includes:

and S402, pairing the first coordinate and the second coordinate by using a pairing search algorithm.

S404, processing the paired result by using an iterative nearest neighbor optimization algorithm, and determining a first registration relation between the fault coordinate system and the binocular camera coordinate system.

The pairing search algorithm is an algorithm for searching for pairing or matching items in a data set. Such algorithms are typically used to find similar items in two data sets or to find paired items in one data set that meet certain conditions. The pairing search algorithm may include: violence search algorithms, hash table algorithms, ordering and double pointers, binary search, super4PCS algorithms, etc., and those skilled in the art may select an appropriate pairing search algorithm to process according to actual situations, and in some embodiments of the present disclosure, the pairing search algorithm is not limited. Iterative nearest neighbor optimization (Iterative Closest Point, ICP) is an algorithm for registration (registration) between point clouds or three-dimensional models. Its main objective is to find the best transformation between two point clouds, minimizing the error between them. The basic idea of the ICP algorithm is to constantly optimize the registration of the point cloud in an iterative manner. Its workflow generally includes the following steps: an initial transformation matrix is randomly selected for use in aligning one point cloud into the coordinate system of another point cloud. For each point, the closest point in the target point cloud is found, which points are called nearest neighbors. And calculating the error between the two point clouds according to the corresponding relation between the nearest neighbors, and updating the transformation matrix according to the error. The above steps are repeated until a termination condition (e.g., error less than a threshold) is met.

Specifically, in general, there may be a plurality of coordinates in the first coordinates and the second coordinates. Therefore, in order to ensure that the corresponding coordinates can be accurately matched, the pairing search algorithm can be utilized to pair various coordinates in the first coordinates and the second coordinates, and further a pairing result is obtained. And then, the paired results can be processed by using an iterative nearest neighbor optimization algorithm, and a first registration relation between the fault coordinate system and the binocular camera coordinate system is determined according to the results output by the iterative nearest neighbor optimization algorithm.

In this embodiment, the accuracy of the first registration relationship can be ensured and the accuracy of the final registration can be improved by using the pairing search algorithm and iterating the nearest neighbor optimization algorithm for processing.

In one embodiment, as shown in fig. 5, the determining the second registration relationship between the tomographic coordinate system and the binocular camera coordinate system and the projection error according to the feature points in the tomographic image and the oral tomographic image includes:

s502, extracting projection two-dimensional features and camera two-dimensional features from the tomographic projection image; the two-dimensional projection features are features of the target position in the three-dimensional data of the oral cavity projected onto the binocular image, and the two-dimensional camera features are features of the target position in the binocular image. The target position may be explained with reference to the above embodiments, and the description thereof will not be repeated here.

Specifically, since two types of images exist in the tomographic projection image, features of the same position (target position) in the two types of images can be extracted, respectively, to thereby obtain a projection two-dimensional feature and a camera two-dimensional feature.

S504, determining a second registration relation between a fault coordinate system and the binocular camera coordinate system based on the projection two-dimensional feature and the camera two-dimensional feature.

Specifically, since the above-described first registration relationship is not very accurate, a more accurate registration relationship is obtained for further use. A second registration relationship between the tomographic coordinate system and the binocular camera coordinate system may be calculated, for example, from the two-dimensional projected two-dimensional features and the camera two-dimensional features.

In some exemplary embodiments, the second registration relationship may be determined in the manner as mentioned in the above embodiments, and other methods may be used to determine the second registration relationship.

S506, calculating to obtain the projection error based on the tomographic projection image, the characteristic points in the oral tomographic image, the characteristic coordinates corresponding to the characteristic points, the parameters of the binocular camera and the second registration relation.

Specifically, after the second registration relationship is obtained, in order to verify whether the second registration relationship is accurate, the projection error may be calculated based on the feature points in the tomographic image and the oral tomographic image, the feature coordinates corresponding to the feature points, the parameters of the binocular camera, and the second registration relationship. Whether the second registration relationship is accurate is determined based on the projection error.

In this embodiment, the accuracy of the final registration can be further improved by determining the second registration relationship in the two-dimensional features, and in addition, by using the projection two-dimensional features and the camera two-dimensional features, the above features are in the tomographic projection image, and the influence of other factors can be reduced.

In one embodiment, as shown in fig. 6, the extracting the projection two-dimensional feature and the camera two-dimensional feature in the tomographic projection image includes:

s602, determining a target position at an image of the oral cavity three-dimensional data projection in the tomographic projection image, and determining a projection two-dimensional feature based on a normal corresponding to the target position.

S604, determining a two-dimensional characteristic of the camera based on the target position in the binocular image in the tomographic projection image.

Where normal (normal) refers to the concept used in geometry to describe the perpendicular direction of a curved or planar surface. For a plane, the normal is the perpendicular vector to the plane, and for a curved surface, the normal is the perpendicular vector at some point on the curved surface. The normal is typically expressed as a unit vector, which indicates the orientation of a curved surface or plane. In the fields of computer graphics, computer vision, and geometric modeling, normal is a very important concept. It can be used to describe the surface orientation of an object for illumination calculation, shadow generation, collision detection, surface reconstruction, etc. In three-dimensional graphics rendering, the normal direction may affect the lighting effect, making the object look more realistic. In three-dimensional modeling and computer aided design, the normal is also used to represent the direction and shape of a surface for surface reconstruction, surface fitting, point cloud registration, and other tasks.

Specifically, since the target position in the three-dimensional data of the oral cavity is projected onto the binocular image, it is projected into the binocular image. The target position projected onto the binocular image may be inaccurate due to errors. To reduce the error, a search may be performed in the normal direction to determine the corresponding target location and thus the projected two-dimensional feature. Under normal conditions, the stepping of the corresponding target position obtained based on normal search is the largest, so that the iteration times can be reduced, and the registration efficiency can be improved. In order to calculate the error, the two-dimensional camera characteristic can also be determined at this time from the target position of the binocular image in the tomographic image. And then a mapping relation can be established according to the projection two-dimensional characteristics and the camera two-dimensional characteristics.

In some exemplary embodiments, three-dimensional data of the oral cavity in a fault coordinate system is projected to a binocular image after coordinate conversion, and a fault projection image is generated; then extracting the two-dimensional contour feature information of the occlusal surface (target position) of the tooth again on the tomographic projection image, searching the two-dimensional contour mapping points of the binocular image at the position in the tomographic projection image again in the normal direction of each point, and establishing a new mapping relation (second registration relation), for example, obtaining the mapping relation of the following formula:

Three-dimensional P of oral data _m Tomographic data projection image two-dimensional feature u _m />Binocular image two-dimensional feature u _c

Wherein due to P _m To u _m Is projected and therefore free of errors. And u is a constant error between two dimensions and three dimensions _m To u _c May not overlap with each other, so u is required to be obtained _m To u _c Mapping relation (second registration relation) between the first and second images.

In general, the mapping relationship (second registration relationship) is a classical PnP problem, and the second registration relationship can be obtained by solving in a DLT (direct linear transformation method), epnP, etc., where the second registration relationship also includes a transformation matrix R and a translation t.

In this embodiment, the number of iterations can be reduced, errors can be reduced, and the registration speed can be increased by the normal direction.

Specifically, the projection error can be calculated using the following formula:

wherein Err isProjection error, n _l Is the number of characteristic points in the two-dimensional characteristic of the camera in the left eye image in the binocular image, n _r Is the number of feature points in the two-dimensional features of the camera in the right eye image in the binocular image. u (u) _cl U is the coordinates of the feature points inside the two-dimensional features of the camera in the left-eye image _cr The coordinates of feature points inside the two-dimensional features of the camera in the right-eye image. P (P) _ml Is equal to u _cl Coordinates of corresponding three-dimensional feature points in the corresponding oral tomographic image. P (P) _mr Is equal to u _cr Coordinates of corresponding three-dimensional feature points in the corresponding oral tomographic image. R is R _rl ,t _rl ,K _l ,K _r Parameters known for binocular cameras. R and t are the second registration relationships.

In this embodiment, by using the above various parameters, the calculation error obtained finally can be more accurate, and the accuracy of registration is further improved.

In one embodiment, the method further comprises:

Specifically, when the projection error is greater than or equal to the preset threshold, it may be generally determined that the determined second registration relationship has a greater error, so that the second registration relationship may be continuously updated iteratively, and since the second registration relationship is already updated iteratively, and the second registration relationship is generally more accurate than the first registration relationship, in order to further reduce the number of iterations of the second registration relationship, the adjusted second registration relationship and parameters of the binocular camera may be used to project the three-dimensional data of the oral cavity onto the binocular image, thereby reconstructing a tomographic projection image, and using the reconstructed tomographic projection image, by adopting the manner mentioned in the above embodiment, the projection error is recalculated, and when the projection error is smaller than the preset threshold, it is proved that the iteratively updated second registration relationship satisfies the requirement, and the iterative updating of the second registration relationship is stopped. Otherwise, continuing to adjust the second registration relationship.

In this embodiment, when the projection error is greater than or equal to the preset threshold, the second registration relationship may be continuously updated in an iterative manner, so as to ensure the accuracy of registration.

In one embodiment, as shown in fig. 7, the embodiment of the present disclosure further provides a method for registering an oral tomographic image, including: the oral cavity CBCT data may be photographed first, and then tooth CT three-dimensional data boundaries in the CBCT data are extracted. The patient may then wear the reference device and take binocular images with the binocular camera. Binocular images typically include: teeth and a reference device. The pose of the reference device may be located based on the reference device in the binocular image. Tooth two-dimensional data boundaries in the binocular image may also be extracted. And then three-dimensional reconstruction is carried out on the tooth two-dimensional boundary data, and coarse registration is carried out on the tooth two-dimensional boundary data after three-dimensional reconstruction and the tooth CT three-dimensional data boundary in a three-dimensional hierarchy (namely, the first registration relation is determined in the embodiment). The oral CBCT data and parameters of the binocular camera may then be mapped into the binocular image according to the first registration relationship, thereby calculating a second registration relationship and projection errors. When the projection error does not meet the threshold value, the second registration relation can be adjusted, the oral cavity CBCT data and the parameters of the binocular camera are mapped into the binocular image again according to the adjusted second registration relation, and the projection error is recalculated until the projection error meets the threshold value. And then registering according to the second registration relation and the pose of the reference device.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiments of the present disclosure also provide an apparatus for registering an oral tomographic image for implementing the above-mentioned method for registering an oral tomographic image. The implementation of the solution provided by the device is similar to that described in the above method, so the specific limitation in the embodiment of the device for registering one or more oral tomographic images provided below may be referred to the limitation of the method for registering oral tomographic images hereinabove, and will not be repeated here.

In one embodiment, as shown in fig. 8, there is provided a registration apparatus 700 of an oral tomographic image, including: an image acquisition module 702, a first relationship determination module 704, an image generation module 706, a calculation module 708, and a registration module 710, wherein:

the image acquisition module 702 is configured to acquire an oral tomographic image of a target object and a binocular image of an oral cavity after a reference device is installed in the oral cavity of the target object, where the oral tomographic image is obtained after computed tomography, and the binocular image is obtained after photographing by a binocular camera.

A first relationship determining module 704, configured to determine a first registration relationship between a tomographic coordinate system corresponding to the oral tomographic image and a binocular camera coordinate system corresponding to the binocular image according to features in the oral tomographic image and the binocular image.

An image generating module 706, configured to project the three-dimensional data of the oral cavity in the tomographic coordinate system onto the binocular image based on the first registration relationship and the parameters of the binocular camera, and generate a tomographic projection image.

A calculation module 708 is configured to determine a second registration relationship between the tomographic coordinate system and the binocular camera coordinate system and a projection error according to the tomographic projection image and the feature points in the oral tomographic image.

And the registration module 710 is configured to determine, based on the second registration relationship and the pose of the reference device in the binocular image, a conversion relationship between a reference coordinate system corresponding to the reference device and the fault coordinate system in response to the projection error being smaller than a preset threshold, and perform registration based on the conversion relationship.

In one embodiment of the apparatus, the first relationship determination module 704 includes:

the first coordinate determining module is used for extracting first features corresponding to target positions in the oral cavity from the oral cavity tomogram and determining first coordinates of the first features in a tomogram corresponding to the oral cavity tomogram.

And the second coordinate determining module is used for extracting a second feature corresponding to the target position from the binocular image and determining a second coordinate of the second feature in a binocular camera coordinate system corresponding to the binocular image.

And the registration relation determining module is used for determining a first registration relation between the fault coordinate system and the binocular camera coordinate system based on the first coordinate and the second coordinate.

In an embodiment of the apparatus, the registration relationship determination module is further configured to pair the first coordinate and the second coordinate using a pairing search algorithm; and processing the pairing result by using an iterative nearest neighbor optimization algorithm, and determining a first registration relationship between the fault coordinate system and the binocular camera coordinate system.

In one embodiment of the apparatus, the computing module 708 includes:

the two-dimensional feature determining module is used for extracting projection two-dimensional features and camera two-dimensional features from the tomographic projection image; the two-dimensional projection features are features of the target position in the three-dimensional data of the oral cavity projected onto the binocular image, and the two-dimensional camera features are features of the target position on the binocular image;

a second relationship determination module configured to determine a second registration relationship between a tomographic coordinate system and the binocular camera coordinate system based on the projected two-dimensional feature and the camera two-dimensional feature;

and the calculating sub-module is used for calculating the projection error based on the tomographic projection image, the characteristic points in the oral tomographic image, the characteristic coordinates corresponding to the characteristic points, the parameters of the binocular camera and the second registration relation.

In one embodiment of the apparatus, the two-dimensional feature determining module is further configured to determine a target position at an image of the three-dimensional data projection of the oral cavity in the tomographic projection image, and determine a projected two-dimensional feature based on a normal direction corresponding to the target position; and determining a two-dimensional camera characteristic based on the target position in the binocular image in the tomographic projection image.

In an embodiment of the device, the calculating submodule is further configured to calculate a projection error according to the number of feature points in the two-dimensional features of the camera in the left-eye image and the right-eye image of the binocular image in the tomogram, coordinates of the feature points, coordinates of three-dimensional feature points corresponding to the feature points in the two-dimensional features of the camera in the oral tomogram, parameters of the binocular camera, and the second registration relationship.

In one embodiment of the apparatus, the apparatus further comprises: and the registration relation updating module is used for responding to the projection error being greater than or equal to the preset threshold value, adjusting the second registration relation, projecting the oral three-dimensional data in the fault coordinate system onto the binocular image by utilizing the adjusted second registration relation and the parameters of the binocular camera, regenerating a fault projection image and calculating the projection error until the projection error is less than or equal to the preset threshold value.

The respective modules in the above-described registration apparatus for oral tomographic images may be realized in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 9. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing data of oral cavity tomographic images, binocular images, first registration relations and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of registration of oral tomographic images.

It will be appreciated by those skilled in the art that the structure shown in fig. 9 is merely a block diagram of a portion of the structure associated with the disclosed aspects and is not limiting of the computer device to which the disclosed aspects apply, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of any of the method embodiments described above when the computer program is executed.

In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the steps of any of the method embodiments described above.

In an embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, implements the steps of any of the method embodiments described above.

It should be noted that, the oral tomographic image and the binocular image related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided by the present disclosure may include at least one of non-volatile and volatile memory, among others. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided by the present disclosure may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors involved in the embodiments provided by the present disclosure may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic, quantum computing-based data processing logic, etc., without limitation thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples have expressed only a few embodiments of the present disclosure, which are described in more detail and detail, but are not to be construed as limiting the scope of the present disclosure. It should be noted that variations and modifications can be made by those skilled in the art without departing from the spirit of the disclosure, which are within the scope of the disclosure. Accordingly, the scope of the present disclosure should be determined from the following claims.

Claims

1. A method of registration of an oral tomographic image, the method comprising:

2. The method of claim 1, wherein the determining a first registration relationship between a tomographic coordinate system corresponding to the oral tomographic image and a binocular camera coordinate system corresponding to the binocular image from features in the oral tomographic image and the binocular image comprises:

3. The method of claim 2, wherein the determining a first registration relationship between the tomographic coordinate system and the binocular camera coordinate system based on the first and second coordinates comprises:

4. The method of claim 1, wherein the determining a second registration relationship and projection error between the tomographic coordinate system and the binocular camera coordinate system from feature points in the tomographic image, the oral tomographic image comprises:

5. The method of claim 4, wherein extracting projection two-dimensional features and camera two-dimensional features in the tomographic image comprises:

6. The method according to claim 4, wherein the calculating the projection error based on the feature points in the tomographic image and the oral tomographic image, feature coordinates corresponding to the feature points, parameters of the binocular camera, and the second registration relationship includes:

7. The method according to any one of claims 1 to 6, further comprising:

8. A registration apparatus for oral tomographic images, the apparatus comprising:

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.