WO2018152742A1

WO2018152742A1 - Multiple surfaces for physical-to-image/image-to-physical registration and image verification

Info

Publication number: WO2018152742A1
Application number: PCT/CN2017/074580
Authority: WO
Inventors: Yu Hang Walter LAM; Tai Chiu Richard HSUNG; Wai Kuen Henry LUK; Ho Nang Edmond POW; Yuk Tung Henry NGAN
Original assignee: Versitech Limited; The Hong Kong Baptist University
Priority date: 2017-02-23
Filing date: 2017-02-23
Publication date: 2018-08-30
Also published as: CN110462681A; CN110462681B

Abstract

This invention is about a novel image registration. This rigid registration is based on multiple surfaces arranged in a 3D relationship in the Cartesian coordinate system through a fiducial marker. In comparison to surface-based registration, this registration process provides more accurate registration or requires a smaller fiducial marker for the same level of registration accuracy. Moreover, this fiducial marker enables measuring coordinates in the physical domain; therefore the registration error can be measured by comparing coordinates of specific points in both physical and virtual domains. This has a potential application in comparing the imaging physical-to-image (PI) or printing image-to-physical (IP) modalities, for example, to compare the accuracies of the 3D model created by computed tomography and optical scanners. Other potential application also includes assisting a surgical navigation in a dental implant placement. The proposed fiducial marker therefore has potential development in the application of virtual reality.

Description

MULTIPLE SURFACES FOR PHYSICAL-TO-IMAGE/IMAGE-TO-PHYSICAL REGISTRATION AND IMAGE VERIFICATION

Field of the invention

This invention relates to image registration based on multiple surfaces arranged in a three-dimensional (3D) relationship as fiducial markers.

Background of the Invention

Virtual images are extensions of the real physical world and the accurate connection between the physical and virtual domains has many potential applications. 3D virtual images of a physical object can be produced by multiple imaging techniques. Non-contact optical scanners are commonly employed to scan objects’surface topography. Multiple 2D images were taken and reconstructed into 3D images (physical-to-image registration, PI) . The accuracy of surface scanning is dependent on the detector of the scanner and the surface properties of the object, such as its optical reflection. Before scanning a particular object, calibration of the optical scanner with objects of similar surface properties and known surface coordinate is recommended. On the other hands, non-reflective powders may be sprayed onto the objects’surface to standardize their surface properties. Its thickness however is difficult to control.

In the medical field, surgeons can acquire 3D computed tomography (CT) images of their patients to investigate the inner parts of human body and plan for any required surgical treatment. A 3D grey-scale image and orthogonal cross-sectional (sagittal, coronal and horizontal) views of the patient’s jaw bone in the selected field of view (FOV) will be generated from a serial of 2D images. Surgeons can plan and place virtual dental implants at a position which will facilitate biological and esthetic tooth replacement in CT images using surgical planning software. Real time computer-assisted navigation (dynamic) or template-based (static) surgical guidance has been developed to facilitate accurate implant placement by transferring this pre-operative virtual planning to the operating table (image-to-physical registration, IP) .

Multiple imaging techniques may be combined together for clinical use. While CT provides 3D images of internal structures of the skull, optical surface scanning provides high resolution 3D teeth surface model as a medium for IP. 3D images may therefore be superimposed to supplement each other (image-to-image registration, II) .

Currently, there are several algorithms commonly used for image registration. For example, the best-fit superimposition (intensity-based) , point-based and surface-based registration algorithm. The best-fit algorithm is used for the II registration by matching the intensity of image elements between 2 images of an imaging technique. However, it does not allow any PI/IP registration conversion, nor II registration among multiple imaging techniques. The best-fit algorithm has a mean error of 0.16 mm (linear) and 1.07 mm (3D) Lagravère, Manuel O., et al. "Three-dimensional accuracy of measurements made with software on cone-beam computed tomography images. " American Journal of Orthodontics and Dentofacial Orthopedics 134.1 (2008) : 112-116. Point-based registration involves a placement of some markers physically on the object. More points will reduce the registration error. See Fitzpatrick et al., “Predicting error in rigid-body point-based registration, ” IEEE Transactions on Medical Imaging 17.5 (1998) : 694-702, which is incorporated herein by reference in its entirety, hereinafter “Fitzpatrick et al. 1998. ” However this will increase the computational power/time needed for registration and may make it practically very difficult to place adequate points in limited space such as the oral cavity. This method has errors of 0.3 mm to detect the central point in a physical domain and 0.4 mm in a CT scan See, Maurer, Calvin R., et al. “Registration of head volume images using implantable fiducial markers, ” IEEE Transactions on Medical Imaging 16.4 (1997) , pp 447-462, which is incorporated herein by reference in its entirety, herein after “Maurer 1997. ” For surface based registration, it was not commonly used in the PI and IP conversion since it requires a large surface for acceptable registration accuracy and the problem of using resilience soft tissues in the registration. The use of optical surface scanner in capturing the surface of an object may encounter problems due to the surface reflective properties of an object and errors may be result in capturing the true surface of an object which is difficult to verify by existing registration methods. See, Ireland, A.J., et al. “3D surface imaging in dentistry–what we are looking at, ” British Dental Journal 205.7 (2008) , pp 387-392, which is incorporated herein by reference in its entirety, hereinafter “Ireland et al. 2008. ”

Ideally, the image registration error should be the minimum and most of the safety margin should be reserved for the surgical treatment itself, such as the vibration of surgical instruments. In placing any dental implant, the usual clinical safety margin would be 2 mm and any registration error in the computer-assisted surgery may already account for half of this safety margin. Moreover, all of these image registrations should be verified physically, yet the arrangement of current point-based/surface-based fiducial markers do not allow easy physical coordinate measurements.

The present inventors achieved promising preliminary results with manual and semi-automatic registration using multiple surfaces with 3D relationships (one cubic corner/corner cube ( “CC” ) of 1 cm ×1 cm ×1 cm) See, Lam, Walter YH, et al. "Validation of a Novel Geometric Coordination Registration using Manual and Semi-automatic Registration in Cone-beam Computed Tomogram. " IS&T Electronic Imaging 2016.14 (2016) : 1-6, which is incorporated herein by reference in its entirety. The disclosed manual and semi-automatic registrations were used to detect the cubic corner.

The Manual registration (MR) was achieved with SimPlant Pro software, i.e., manual registration was used to register the (x, y, z) -axes to three orthogonal windows in SimPlant Pro. Semi-automatic registration (SR) was used in MeshLab and 3D slicer (both free and open-source software) to define each surface by using MeshLab first and then fitting these surfaces to the (x, y, z) -axes in 3D slicer.

Registration errors were found to be 0.56 mm and 0.39 mm for the manual and semi-automatic registrations, respectively. This work anticipated that the registration error could be further reduced if this registration could be performed automatically. Moreover, this registration allows easy physical measuring of the coordinates of specific points. This facilitates comparison with the virtual domains and the error of this registration process can be calculated. This registration process also allows selection of better image modalities (e.g. different types of optical scanners) by comparing their PI registration errors. It was assumed that this technique has the potential to be converted into an ISO standard for measuring the accuracy of image modalities/systems. This may also be potentially useful in measuring the printing (IP) quality of 3D printers in the future. Clinically, by comparing pre-and post-operative serial images, clinicians can assess the accuracy of various surgical navigation or guidance methods.

Summary of the Invention

The present invention concerns the use of multiple surfaces (e.g. three orthogonal surfaces forming a corner of a cube) for serial image registration and calibration. These surfaces have an intrinsic relationship which defines the origin and the x-y-and z-axes of a Cartesian coordinate system. Physically, these surfaces are fitted onto an object such as a patient wearing a surgical appliance/template. This allows physical coordinate measurement of a particular point of an object by a coordinate measuring machine (CMM) . In a computer environment, the 3D image of the scanned surfaces is registered to Cartesian coordinate using these surfaces. Therefore the object position in the physical and virtual computer environments can be matched to the same coordinate system. By comparing coordinates of selected points, the registration error (target registration error TRE) of this method can be identified. With the present invention an automatic means uses the cubic corner (CC) to define the Cartesian coordinates with a corner tip as the origin O and the three line angles as (x, y, z) -axes. This automatic registration reduces human errors.

With the present invention automatic registration is achieved with stereotactic/fiducial markers that are identifiable in both CT and optical surface scanning by its surface smoothness+/-color Red-Green-Blue (RGB) +/-radiopacity. The recognition can be done in a computer incorporated with MATLAB software. The accuracy of registration is increased by adding two flat surfaces distant to CC, which in effective increase the area for registration while allow smaller sized fiducial markers to be used.

With knowledge of the registration error of the multiple surfaces, this method can be extended to measure the accuracy of imaging techniques (i.e. PI) such as computed tomography and optical scanners. Therefore, this method allows for a direct comparison of the virtual domain to the physical domain and allows for calibration of imaging equipment. Moreover, this method defines the reference coordinate of an object and allows for comparison of changes in serial images. Thus, it may be used to compare the progress of a disease and evaluate the treatment outcome such as the surgical navigation or guidance.

This structure can be achieved by modifying

bricks ( “flat tiles” or “wall corners” ) to act as fiducial markers. The CC helps to define the Cartesian coordinates both in the physical domain (real patient/object) as well as in the multiple imaging domains, such as CT and optical surface scanner. This process is referred to as “registration. ” Therefore, both physical and image will share the same coordinates in a perfect registration.

The current invention proposed a new algorithm to register fiducial markers (surfaces) on appliances, including templates and in serial images automatically and to reduce error in IP, II and PI registrations. A dental surgical appliance may be a tooth-supported device that fits over teeth and guides the oral surgeon in correct placement of implants in patients’mouth. They are usually made with acrylic, polypropylene or similar materials. The proposed surfaces on the appliance/stent define the Cartesian coordinate in the physical and virtual domains and allow the verification of IP, II and PI registration (TRE) . Individual imaging and/or printing technique can be validated and compared to other techniques.

Multiple principal surfaces (1 cm ×1 cm) are arranged in a 3D orthogonal relationship physically or virtually. Physically, this can be a machine-milled cubic corner or commercially available cubic blocks such as

brick’s “flat tile” . Alternatively, a cubic corner space (an inverse cubic corner) may be milled or again using commercially available

brick’s “wall corner” . The inverse cubic corner may be useful for locating the tip of a surgical instruments and useful in IP. Virtually, this can be easily designed by computer aided design (CAD) software.

Two supplementary geometric structures, such as flat surfaces, may be formed with a circular/oval shape (centroid) with a diameter of 0.5 cm. Physically, they are custom-made by milling a flat plate, e.g., one made of acrylic or commercially available

brick’s “flat tile” . Alternatively, two “wall corners” may be used. Virtually, this can again be easily designed by software.

Golden ratio/triangular/geometric algorithm may be included between i) the origin of the principal surfaces (corner tip) and two supplement centroids or ii) three surfaces forming the principal fiducial marker. Physically, there are hand instruments (e.g. a golden ratio caliper) to facilitate their positioning. Virtually, this can also be easily designed by software.

For a PI registration, depending on the nature of the imaging modalities, these physical surfaces can be smoothened (for optical scanner and computed tomography) , color-painted (for optical scanner) or painted with a thin film of radiopaque material (for computed tomography) . Algorithm for detecting the surfaces may be based on surface smoothness, color (e.g. RGB color) , and radiopacity (grey scale) .

In comparison to a surface-based registration of similar area, this registration process provides more accurate registration. In other words, a smaller fiducial marker is needed for the same level of registration accuracy. Moreover, this fiducial marker enables measuring coordinates in the physical domain； therefore the registration error (TRE) can be measured by comparing coordinates of specific points in both physical and virtual domains. This has a potential application in comparing the imaging (PI) /printing (IP) modalities. For example, to compare the accuracy of 3D images created by computed tomography and by optical scanners.

Currently, there are several algorithms commonly used for image registration. For example, the best-fit superimposition (intensity-based) , point-based and surface-based registration algorithm. The best-fit algorithm is used for the II registration by matching the intensity of image elements between 2 images. However, it does not allow any PI or IP registration conversion. The best-fit algorithm has a mean error of 0.16 mm (linear) and 1.07 mm (3D) See Lagravere et al. 2008. Point-based registration involves a placement of some markers physically on the object. More points will reduce the registration error. See Fitzpatrick et al., 1998. However, this will increase the computational power/time needed for registration and may practically difficult to place adequate points in limited space such as the oral cavity. This method has errors of 0.3 mm to detect the central point in a physical domain and 0.4 mm in a CT scan. See, Maurer 1997. For surface based registration, it was not commonly used in the PI and IP conversion and it requires a large surface for acceptable registration accuracy and the problem of using resilience soft tissues in the registration. The use of optical surface scanner in capturing the surface of an object may encounter problems due to the surface reflective properties of an object and errors may be result in capturing the true surface of an object which is difficult to be verified by existing registration methods. See, Ireland 2008.

Ideally, the image registration error should be the minimum and most of safety margin should be reserved to the surgical treatment itself, such as the vibration of surgical instruments. In placing any dental implant, the usual clinical safety margin would be 2 mm and any registration error may already account for half of the safety margin. Moreover, all these image registrations should be verified physically, yet the arrangement of current point-based/surface-based fiducial markers do not allow easy physical coordinate measurements.

Brief description of the drawings

The foregoing and other objects and advantages of the present invention will become more apparent when considered in connection with the following detailed description and appended drawings in which like designations denote like elements in the various views, and wherein:

FIG. 1 shows a cubic corner fiducial marker according to the present invention with three surfaces arranged in orthogonal relationships matched to the x-y-z-axes of Cartesian coordinates；

FIG 2A shows automatic surface detection by surface roughness and FIG. 2B shows automatic surface detection by color；

FIG. 3A shows a CT image of milled multiple surface fiducial markers (one cubic corner and two flat surfaces) on an oral appliance/stent sitting on the maxillary (upper) arch of a patient and FIG. 3B shows an automatic software determined image of the fiducial marker surfaces bytheir smoothness for image registration；

FIG. 4A shows commercially available

brick “flat tiles” used as fiducial markers on the oral appliance, FIG. 4B is a CT image of a top plan view of the and “wall corners” markers and appliance, and FIG. 4C is a computer generated schematic diagram of the markers and their positioning；

FIG. 5A is a software generated image of a milled principal cubic corner and two centroid supplement surfaces arranged in a triangular algorithm for image registration according to the present invention, FIG. 5B is an example of triangular/golden triangle ratio algorithm, FIG. 5C is an example of principal fiducial marker in which three centroid surfaces arranged with both orthogonal and triangular algorithm relationship； and

FIG. 6 shows the positioning of milled flat surfaces, i.e., “flat tiles, ” on an oral appliance/stent using a surface plate.

Detailed Description of Exemplary Embodiments

The present invention is an automatic rigid image registration process using multiple geometric surfaces arranged in certain 3D relationships as fiducial markers. The principal arrangement uses three surfaces arranged in an orthogonal relationship, e.g. a cubic corner. The proposed registration process is a modification of surface-based registration and the 3D relationships between these surfaces allow more accurate registration than the surfaces alone. Therefore, a smaller fiducial marker can be used with the same level of registration error as the current ones. This rigid registration process is suitable for image-to-physical (IP) , image-to-image (II) and physical-to-image (PI) registration.

A multiple (principal) surfaces fiducial marker can be attach to an object either physically or virtually, and will define the Cartesian (x, y, z) -coordinates for that object in both physical and virtual domains. The (x, y, z) -coordinates of a point in the object can be measured by both a coordinate measuring machine (CMM) physically and software virtually. The proposed registration process bridging the virtual and physical domains since they can verify each other by a comparison of coordinates measured virtually and/or physically.

The algorithm detects the surface roughness, color or radiopacity of selected surfaces and automatically fits them to (x, y, z) -axes in the Cartesian coordinate system. In particular, a computer program will automatically detect these surfaces and minimize manual error in the registration process.

In deriving the algorithm the following assumptions are made:

A1. Registration in physical domain can be compared with that in virtual domain.

A2. Registration at markers A, B, C induces registration error.

A3. Registration error≤Clinical error

Definition 1: A rectangular marker in the real domain (RD) is defined as M＝ [f (x, y) ] and a rectangular marker in virtual domain (VD) ,

is defined as

where

and P＝ [e (x, y) ] is a m×n matrix that represents additive clinical errors, and e (x, y) ∈R,

1≤x≤m, 1≤y≤n.

Definition 2: Given any two m×n markers, M^v＝ [f^v (x, y) ] and M^r＝ [f^r (x, y) ] , a k-norm metric between two markers is defined as

where N＝m·n.

Remark: For a circular marker in the RD with a radius r and a centroid c＝f (x_c, y_c) , it can be defined as Q＝ [f (x, y) ] , where f (x, y) ＝0 if | f (x, y) -c| >2r for a diameter 2r. A circular marker in the VD,

is defined as

where

and P＝ [e (x, y) ] . This can be regarded as a circular disc embedded in a square that any pixel outside the disc is equal to zero.

Definition 3: Given any m x n marker

An ordered sequence of all elements of M is

A shift operation H of a matrix produces m×n new markers such that

is defined as

Given M^r, and

where

is one of the shift matrices, the 1-norm metric between them is defined as

for any two markers M^v,

where

Definition 4: Given any two m×n markers M^v＝ [f^v (x, y) ] and M^r＝ [f^r(x, y) ] , there exists a set of shift markers,

and m·n 1-norm metrics

for M^v, M^r, and an Energy of Shift Difference between M^v and

is defined as

where 0≤K_v, r≤1.

Definition 5: Given three markers M^A, M^B, M^C in the RD and three markers

in the VD, there are several types of errors in registration from the RD to the VD:

(a) Translational error in x, y, z-axes:

E_t＝e_A+e_B+e_C

where e_A, e_B, e_C are from the markers M^A, M^B, M^C, respectively.

(b) Angular error in x, y, z-axes:

where e_A, e_B, e_C are from the markers M^A, M^B, M^C, respectively.

Theorem 6 (Eligible registration errors) : Given three proposed markers M^A, M^B, M^C in the RD, three markers

in the VD such that

where

respectively for translational and angular errors, e_A (x, y) , e_B (x, y) , e_C (x, y) .

Meanwhile, an 1-norm metric between the two markers as follows:

where M^R and

are markers in the RD and VD, respectively. It can be define that

(1) The range of 1-norm metric

between RD and VD markers is

where

is 1-norm metric between two markers.

(2) For e≠0, there exists a tolerance of registration errors, σ_ij, for the 1-norm metric. For σ_ij≥0, the difference in the 1-norm metric between markers in the RD and RD is in a range of

where |e| ＝ |e^v| , e^v∈R.

Remark: In the present case, the eligible registration error σ_ij＝0.02mm and the eligible clinical error e＝2mm. Therefore, σ_ij＝0.02mm≤measurement≤e＝2mm.

Proof: For (i) , there is one case considering the range of difference in 1-norm metric between markers of M^R and

where

is shifted.

Case 1: For e^v (x, y) >0, (a0) becomes

By the triangle rule, |a+b| ≤ |a| + |b| , we have

Case 2: For e^v (x, y) <0, (a0) becomes

Since |x-y| ≥ | |x| - |y| | for all real x, y, we have

Case 3: For e^v (x, y) ＝0, it is trivial that

Combining the results from all these cases,

where

1≤x, i≤m and 1≤y, j≤n.

Tolerance of Registration

Since

is equivalent to

for every e≠0, there exists a tolerance of registration errors for the 1-norm metric, σ_ij≥0 such that the difference in 1-norm metric between M^R and

are markers in the RD and VD, respectively, is in a range

An automatic registration algorithm to these orthogonal surfaces reduces any manual errors in image registration. In cone-beam computed tomography (CBCT) , the PI registration error was found to be less than 1 mm and this accuracy is comparable to current point based registration.

Definition 7 (An error restriction by the golden triangular ratio) :

Given three markers M^A, M^B, M^C in the RD, three markers

in the VD, there are three corresponding centroid points f_A (x_c, y_c) , f_B (x_c, y_c) , f_C (x_c, y_c) in markers M^A, M^B, M^C in the RD and centroid points

in markers

in the VD. Then, there are three corresponding sets of energies,

In the RD, if the Euclidean distances between each pairwise centroids are D_A, D_B, D_C and the corresponding angles at α, β, γ, then two golden triangular ratios are derived by the Law of Sines as

and the Law of Cosines as

D_A ²＝D_B ²+D_C ²- (2D_BD_C) (cos α)

D_B ²＝D_A ²+D_C ²- (2D_AD_C) (cos β)

D_C ²＝D_A ²+D_B ²- (2D_AD_B) (cos γ)

The Laws of Sines and Law of Cosines are valid in the VD as well. These two triangular golden ratios are patient-oriented so that they can be used to measure the registration errors.

Modifications of the principal arrangement

For a surface-based registration, the use of a fiducial marker of larger surface area will increase the registration accuracy. Where there is limited space for positioning of the principal fiducial marker, such as in the oral cavity, two smaller (supplement) geometric structures such as circular/ovoid surfaces (centroids) /wall corners (corner tips) may be added and they should be placed at the same plane/level or at least in parallel to one principal surface. The use of a golden ratio/triangular/geometric algorithm in the positioning of these two centroid/corner tip surfaces and the principal surface is recommended to further enhance the registration accuracy. Alternatively, the principal fiducial marker may be modified by arranging three centroid surfaces in an orthogonal relationship and using the golden ratio/triangular algorithm to determine the space between their centroid positions FIG. 5C. The golden ratio/triangular algorithm is shown in FIG. 5B.

Multiple principal fiducial markers (i.e. a cubic corner) can be used in a rigid registration. However according to the present invention only one marker is selected to define the Cartesian coordinates. The multiple cubic corners may be useful for a surgical navigation in which the virtual surgical planning is transferred to a real physical patient that requires a precise calibration of the position of the surgical equipment (e.g., a drilling bur) between the virtual and physical domains. The inverse corner/wall corner provides one solid point in this calibrating process.

Two perpendicular surfaces are sufficient for the proposed registration process. The third surface can be extrapolated from the two surfaces, substituted by a Cartesian point (for example to define the origin) , or by other means.

A cubic corner can be used in i) Image registration and ii) Linking the virtual and physical domains together. While image registration is based on multiple surfaces that are arranged in a 3D relationship (such as a geometric structure like a sphere) , linking the virtual and physical domains is the unique function of a cubic corner which defines the Cartesian coordinates for both the physical and virtual domains.

The present invention can be used in a surgical navigation or guided surgery procedure. In order to do so the conventional optical scanners/computed tomography typically involved in such a procedure are used. Also, these scanners are connected or the image files are transferred to a computer which runs programs that allow for preoperative surgical planning. In order to make use of the present invention, the computer runs an additional software module that handles the automatic registration based on the scanner’s images captured by the computer. The process involves the following steps:

(1) In the physical domain, fiducial markers may be custom made by milling or commercially available

bricks may be used. In the virtual domain, these markers can be easily designed using proper software. One of the markers is a cubic corner 10 as shown in FIG. 1 and FIG. 2B.

(2) These markers are attached to an oral appliance/stent 12 which sits reproducibly on the patient’s teeth 14. Positioning of markers on the appliance/stent is critical for image registration and linking of the virtual and physical domains together. The cubic corner and two flat surfaces are positioned in one plane by a machine milled surface plate as shown in FIG. 6. Furthermore they may be arranged in a relationship, such as the golden ratio, triangular and other geometric algorithms as shown in FIGS. 5A and 5B, by using a golden ratio caliper or a rectangular Lego block etc. The marker surfaces are polished as shown in FIG. 2A or painted as shown in FIGS. 1 and 2B to aid in automatic identification by the MATLAB software. For detection in Computed Tomography (CT) , the paint is of different radiopacity than the markers, while for the optical surface scanning, colors such as red, green and blue may be painted on the marker surfaces. The automatic alignment can be done in a standalone computer using the acquired data from adopted scanning device (s) . Alternatively, the markers and the oral appliance/stent can be designed virtually and then printing out (stereolithography) . These markers can be easily positioned virtually by the software.

(3) For an imaging approach, the oral appliance with fiducial markers is digitalized and automatically identified by the registration software. The accuracy of registration is increased by adding two flat surfaces or tiles 16 at distances from the cubic corner. This is shown in both a CT scan and optical surface scan images in FIGS. 3A and 3B. A prototype in the form of

flat tile 16 is shown in FIG. 4A, and optical surface scans of

wall corners 15 is shown in FIG. 4B. The position of these “wall corners” 15 can be guided by a rectangular Lego brick (an example of a geometric algorithm) is shown in FIG 4C. The cubic corner helps to define the Cartesian coordinates both in the physical domain (real patient) as well as in the multiple imaging domains, such as CT and optical surface scanner. The coordination process is termed “registration. ” Therefore, both physical structure and the image will share the same coordinates when there is perfect registration.

FIG. 5A shows a CT image of the modified stent of the present invention, i.e., a model image with two centroid supplement discs and one milled principal cube corner whose arrangement is modified by a triangular algorithm of the present invention to solve the serial images registration (detect and reconstruct) problem automatically and to reduce error in the 3D virtual images. FIG. 5B is an example of the triangular/golden triangle ratio algorithm. The corner tip of the cubic corner may function as a centroid and the distances between the corner tip/centroids A, B and C in this invention incorporate the triangular algorithm, e.g., by having AB equal to AC and the ratio of AB (or AC) : BC is 1.6181 to 1. By adopting this relationship between centroids, the size of the fiducial marker can be reduced to a minimum while maintaining a similar degree of registration accuracy. The lengths of AB, AC, BC are all flexible and may be in the golden triangular ratio in both the physical and virtual domains. FIG. 5C is an example of multiple surfaces arranged in orthogonal relationship and their centroids arranged in triangular ratio.

(4) Clinicians then design the plan for the surgical procedure virtually.

(5) For a printing approach, the virtually designed oral appliance can be 3D printed.

(6) During the surgical operation, the physical surgical instruments can be calibrated to link both physical and virtual domains by the fiducial markers, such as the cubic corner or wall corners. The wall corners in FIG 4B and 4C allow the calibration of a surgical instruments to the virtual planning domain. During the calibration, the tip of the surgical instruments can be placed in the corner and its position captured by the tracing system and this can relate the physical instruments to the virtual imaging. Therefore, the virtual surgical planning can be executed (image-to-physical registration) in the real physical world.

(7) Post-operatively, the patient can wear the oral appliance/stent. Then they can be digitalized again. The image registration allows an image-to-image (II) comparison to demonstrate the surgical result. Since the physical and virtual domains are linked together, patients may avoid any post-operative CT, using only optical surface scanning to determine the surgical result (e.g. position of implant) and to compare the preoperative planning in the CT with reference to the CC.

Verification of the registration accuracy and instrument calibration

The proposed fiducial markers can be attached to a testing object. Verification of the registration (PI, II and IP) accuracy and instrument calibration can be performed by comparing the Cartesian coordinates of specific points obtained by a physical CMM and by a virtual software.

The imaging/printing instruments can be calibrated with this process and potentially act as an ISO standard for both the imaging and printing modalities. Selection of more accurate imaging/printing instruments is possible by a comparison of their physical-virtual errors (target registration error TRE) .

While the present invention has been particularly shown and described with reference to preferred embodiments thereof； it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

Claims

A method of registering or aligning a physical object in a Cartesian coordinate system with a virtual three dimensional image, comprising the steps of:

locating a cubic corner on the physical object as a fiducial marker, said cubic corner having one of smooth surfaces and surfaces painted with different colors/radiopacities, at least two of the surfaces of the cubic corner being orthogonal to each other；

scanning the physical object to capture an image of the physical object and the surfaces of the cubic corner；

automatically recognizing the surfaces of the cubic corner in the image based on one of their smoothness and color/radiopacity； and

aligning virtual Cartesian coordinates with the captured surfaces of the cubic corner in the image.
A method of image to image registration or alignment, wherein the image is selected from serial images of an object at different points in time in a Cartesian coordinate system, said method comprising the steps of:

locating a virtual cubic corner in each image as a fiducial marker, said cubic corner having one of smooth surfaces and surfaces painted with different colors/radiopacities, at least two of the surfaces of the cubic corner being orthogonal to each other；

automatically recognizing the surfaces of the cubic corner in each image based on one of their smoothness and color/radiopacity； and

aligning the serial images according to the cubic corners in each image.
A method of forming a three-dimensional object based on a three-dimensional image in a Cartesian coordinate system, comprising the steps of:

locating a virtual cubic corner in the image as a fiducial marker, said virtual cubic corner having one of smooth surfaces and, if the image is original from a physical object, surfaces painted with different colors/radiopacities, at least two of the surfaces of the virtual cubic corner being orthogonal to each other；

automatically recognizing the surfaces of the virtual cubic corner in the image based on one of their smoothness and color/radiopacity； and

one of milling or 3D printing the physical object based on the image so that a cubic corner in the physical object is aligned with virtual Cartesian coordinates in the image,

whereby the cubic corner in the physical object can align with physical Cartesian coordinates in the Coordinate Measuring Machine (CMM) .
The method of claim 1 wherein one of the surfaces being orthogonal to each other is a principal surface and further including the steps of

adding two flat surfaces or tiles to the physical object at distances from the cubic corner prior to the step of scanning, said tiles being located in the same plane as a principal surface of the cubic corner,

positioning of the two flat surfaces or tiles and the principal surface by using the golden ratio/triangular/geometric algorithm；

automatically recognizing the flat surface of the tiles and cubic corner； and

using the location of the tiles and cubic corner to aid in the aligning of the virtual Cartesian coordinates.
The method of claim 4 wherein the flat surfaces or tiles have surfaces with a centroid, such as circular/ovoid-shaped, or the surface of another wall corner/cubic corner.
The method of claim 1 wherein the cubic corner has smooth surfaces and the step of scanning is carried out with a computed tomography (CT) scanner or an optical surface scanner.
The method of claim 2 wherein the virtual cubic corner has smooth surfaces and the computer software that detects surface roughness.
The method of claim 1 wherein the cubic corner has colored/radiopacity surfaces and the step of scanning is carried out with an optical surface scanner that detects Red-Green-Blue colors/computed tomography that detects radiopacity.
The method of claim 2 wherein the virtual cubic corner has colored/radiopacity surfaces and the step of scanning is carried out with computer software that detects color or radiopacity in the image.
The method of any one of claims 1 to 3 wherein the step of aligning is performed using the golden ratio/triangular/geometric algorithm between the centroid/corner tip position.
The method of any one of claims 1 to 3 wherein the tip of cubic corner may function as a centroid in the registration process. Centroid may also come from one principal surface.
The method of claim 1 wherein the cubic corner is located on the physical object.
The method of claim 1 wherein the physical object is an oral appliance or stent.
The method of claim 1 wherein the cubic corner has only two orthogonal surfaces and a third orthogonal surface is extrapolated from the two surfaces.
The method of clam 1 wherein the cubic corner is made by machine milling a flat acrylic resin and polypropylene plate.
The method of claim 2 wherein the surfaces of the cubic corner and flat surface tile are
brick flat tiles.
The method of claim 1 wherein the surfaces of the cubic corner include
brick wall corners.
The method of claim 3 wherein the printing is an additive manufacturing method.
The method of claim 3 wherein the milling is subtractive from a block.
A method for planning a guided navigation surgical procedure comprising the steps of:

scanning the body part to form a virtual model of a body part by computed tomography；

designing a plan for the surgical procedure virtually on the image,

locating one or more wall corner fiducial markers on the virtual model, said wall corner having an origin which provide a positive stop for the physical surgical instruments；

milling or printing the fiducial marker if it is original from virtual model； and

during the surgical guidance operation, the physical surgical instruments are calibrated to both physical and virtual domains by the fiducial markers.
The method of claim 19 further including the step of adding two accessory wall corners to the model at distances from the principal wall corner, said accessory wall corners being located in the same plane as a surface of the principal wall corner,

positioning of the two accessory wall corners cube and the principal wall corner by using the golden ratio/triangular/geometric algorithm；

automatically recognizing the flat surfaces of the cubic corners； and

using the location of principal cubic corner to aid in the aligning of the virtual Cartesian coordinates.
The method of claim 19 wherein the surgical procedure is a dental procedure and the model is an oral appliance sitting on the patient’s teeth.
A method for evaluating the result of a surgical procedure comprising the steps of:

forming a virtual model of a body part and the fiducial marker (s) by computed tomography or surface scanning after the surgery, said cubic corner/wall corner having one of smooth surfaces, at least two of the surfaces of the cubic corner/wall corner being orthogonal to each other；

automatically recognizing the surfaces of the cubic corner/wall corner in the image based on one of their smoothness and color/radiopacity；

aligning virtual Cartesian coordinates with the captured surfaces of the cubic corner/wall corner in the image； and

comparing the interested surgical results with the planning in pre-operative computed tomography.
A method for evaluating the accuracy of the imaging/printing modalities comprising the steps of:

locating a cubic corner/wall corner fiducial marker on the reference model；

forming a test model by scanning/printing of the physical/virtual reference model and the fiducial marker, said cubic corner/wall corner having one of smooth surfaces, at least two of the surfaces of the cubic corner/wall corner being orthogonal to each other, if scanning a physical object, surfaces painted with different colors/radiopacities；

automatically recognizing the surfaces of the cubic corner/wall corner in the image based on one of their smoothness and color/radiopacity；

aligning physical/virtual Cartesian coordinates with the surfaces of the cubic corner/wall corner； and

comparing the coordinate of selected points of the physical/virtual domains.