WO2019164093A1

WO2019164093A1 - Method for improving ct data and optical data matching performance, and device thereof

Info

Publication number: WO2019164093A1
Application number: PCT/KR2018/013581
Authority: WO
Inventors: 신영길; 김동준; 송영찬
Original assignee: 서울대학교산학협력단
Priority date: 2018-02-23
Filing date: 2018-11-09
Publication date: 2019-08-29
Also published as: KR102099415B1; KR20190101694A

Abstract

The present invention relates to a method for improving CT data and optical data matching performance, and a device thereof and, more specifically, to: a method using CT data and optical data so as to perform data matching between both types of data when three-dimensionally reconstructing internal/external structures of an object, thereby reducing the time required for matching, reducing processing complexity, and improving the accuracy of match results; and a device thereof.

Description

Method and device for improving matching performance of CT data and optical data

The present invention relates to a method and apparatus for improving the matching performance of CT data and optical data, and more particularly, to perform data matching between CT and optical data by reconstructing the internal and external structures of an object in three dimensions. The present invention relates to a method and an apparatus capable of reducing the time required for matching, reducing processing complexity, and improving the accuracy of matching results.

In general, CT (Computed Tomography) is an image that provides information about the structure and tissue state of a product that cannot be obtained by a conventional X-ray apparatus. It is a combination of images obtained by projecting a target product by X-ray from various angles. It is a technology to realize a three-dimensional image.

Such CT data has the advantage that the internal and external structures can be reconstructed in three dimensions, but due to the physical properties of the X-ray, the surface is not evenly reconstructed according to material and structural complexity.

In addition, optical data has an advantage that the three-dimensional surface to be reconstructed compared to CT data is evenly expressed and has high precision, but only the surface of the light reflection area exposed to the scanner can be reconstructed.

On the other hand, by reconstructing the internal and external structure of the object in three dimensions by using the advantages of CT data and optical data, it is possible to diagnose lesions using the internal and external information of the object, pre-operative simulation (Image-Guided Surgery), surgical guide using augmented reality It can be used in various ways together.

However, although the conventional CT data and the optical data have their respective advantages, there are some limitations when performing matching.

For example, when extracting the surface from the CT data for matching, non-linear attenuation occurred according to the characteristics and distance of the material through which X-rays pass due to X-ray beam hardening. Photon starvation caused bands, streaks, white lines, shadows, or artifacts to reconstruct the screen model of the CT data.

In addition, due to insufficient scan image or low resolution scan image, a partial volume effect (PVE), in which the boundary part is blurred, is generated when reconstructing the screen model of the CT data. Due to the metal noise caused by the prosthesis, there was a problem that the available surface information is very limited.

As described above, in addition to the limitation of extracting the surface from the CT data, the following limitations exist when the ICP (Iterative Closest Point) algorithm is applied in the matching process between the CT data and the optical data. In this case, the ICP algorithm calculates scale transformation, rotation, and movement to match the measurement data to the model when there is measurement data for a model.

That is, the conventional ICP algorithm uniformly defines the surface vertex data derived without considering the characteristics of the vertex data including the error of the CT data and the optical data in the process of matching the CT data and the optical data. The matching process was very inefficient, as well as the results reflecting the error, as well as computing the vertices that do not need to be considered when performing the algorithm, or disturbing convergence due to outliers.

Therefore, in the present invention, by extracting the vertices with high reliability in the process of matching the CT data and the optical data using the ICP algorithm, and performing matching through sampling of the extracted vertices, the surface vertex information of each of the CT data and optical data This paper proposes a method to reduce the time while increasing the accuracy of the reconstruction results of the 3D model without using all of them.

Next, the prior art existing in the technical field of the present invention will be briefly described, and then the technical matters to be made differently from the prior art will be described.

First, Korean Patent No. 1785326 (2017.09.29.) Relates to a procedure guide information providing system using a surgical guide for dental implants. CT images and oral scan images of the inside of the oral cavity of the subject are obtained, A first step of obtaining a three-dimensional integrated image through registration of the CT scan image and the oral scan image; A second step of selecting and virtually arranging a fixture corresponding to a predetermined implant placement position based on the obtained three-dimensional integrated image; The virtual drill image extracted from the digital drill library is superimposed on the virtually arranged fixture, and the overlapped volume is calculated from the alveolar bone and the fixture from which the portion corresponding to the virtual drill image is removed is compared with the optimal combined volume. A third step of selecting a procedure drill image accordingly; And a surgical guide having a guide hole for guiding the drilling of the procedure hole is formed on one surface of the fixed groove surrounding the inside of the mouth of the subject according to the 3D integrated image, and the procedure drill image is matched for each guide hole. And a fourth step of outputting the displayed drilling report.

However, the prior art merely has a substrate for obtaining a three-dimensional integrated image by matching the CT scan image and the oral scan image, and the surface vertices extracted when the CT data and the optical data proposed in the present invention are matched. The technical difference is obvious because it does not describe a technical configuration that can reduce accuracy and time by using only reliable vertices without using all the information.

In addition, Korean Patent Application Publication No. 2013-0098531 (September 5, 2013) relates to a method of reducing metal artifacts of Citi, and reconstructs the projection image into a stereoscopic image and the first step (S100), which is a process of acquiring a projection image by Citi. A second step (S200) which is a process of performing a process, a third step (S300) which is a process of matching a reconstructed image and a scan data image, and a fourth step which is a process of checking three-dimensional coordinates of the metal region in the reconstructed image Technical features include (S400).

However, in order to solve the problem that surface information including non-uniform noise is derived and do not need to be considered, or to solve convergence caused by outliers, only some reliable vertices are processed without using all surface vertex information. The technical configuration of the present invention is clearly different from the technical features when compared with the above-described prior art in which only a description of the contents for performing matching between the CT image and the scan data image is presented.

That is, the above prior arts suggest a configuration of acquiring a three-dimensional integrated image through registration of a CT photographed image and an oral scan image, and a configuration of performing a registration of a CT image and a scan data image. There is no specific description of the configuration of reconstructing the three-dimensional model without having to use all the surface vertex information by extracting vertices from the CT data and the optical data, and performing matching through sampling of the extracted vertices. No technical implications are evident since they do not suggest or suggest.

The present invention has been made to solve the above problems, by utilizing the advantages of the CT data and the optical data method for improving the matching performance of the CT data and the optical data to reconstruct the internal and external structure of the object in three dimensions and its It is an object to provide a device.

In addition, in the present invention, when the CT data and the optical data are matched, the matching is performed using only some reliable vertices without using all the surface vertex information, thereby reducing the time required for matching and reducing the complexity of the process. Another object is to provide a method and apparatus for improving the matching performance of CT data and optical data that can improve the accuracy of matching results.

In addition, the present invention improves the matching performance of the CT data and the optical data when reconstructing the three-dimensional model of the internal and external structure of the object, to derive surface information containing heterogeneous noise, and to calculate the vertices that do not need to be considered, Another object of the present invention is to provide a method and apparatus for improving the matching performance of CT data and optical data, which can solve the problem of disturbing convergence due to an outlier.

In addition, the present invention improves the matching performance of CT data and optical data to construct a three-dimensional model with high accuracy within a short time, thereby efficiently performing lesion diagnosis, preoperative simulation, and surgical guide using augmented reality using internal and external information of an object. Another object of the present invention is to provide a method and an apparatus for improving the matching performance of CT data and optical data, which can be performed by using the same method.

According to one or more exemplary embodiments, a method of improving matching performance between CT data and optical data includes: an initial matching step of performing initial matching by applying a plurality of corresponding points to CT data and optical data in the matching performance improving device; The surface vertex candidate extraction step of extracting the surface vertex candidates of the CT data and the optical data which has been performed, calculates an error measure of the extracted surface vertex candidates, and for each surface vertex candidates of the CT data and optical data Performing a vertex sampling based on the calculated error value to remove invalid vertices, a matching result calculating step of calculating a matching result based on the sampling result, and performing matching based on the calculated final matching result And a matching performing step of reconstructing the three-dimensional model.

In the initial matching step, the initial matching may be performed by applying three pairs of corresponding points to the CT data and the optical data.

In the vertex sampling step, for the CT data, Measure 1 is calculated based on a CT value and Measure 2 representing a curvature determined according to a geometric shape of a point set around a corresponding position. The noise is removed, but it is determined that noise has a large change in ambient intensity at each position using Measure 1 and a large change in curvature at each position is measured using Noise Measure 2 as noise, and in the case of the optical data, By using Measure 2, it is determined that the change in the curvature of the surroundings is large at each position as noise and is removed. In this case, the reason for determining the vertex with large curvature as noise by using Measure 2 at the optical data surface vertex is that the larger the curvature is, the more difficult the scanner can accurately scan the portion.

The matching result calculating step may also include a matching point calculation step of calculating a corresponding point of the optical data surface vertex with respect to the surface vertex candidate of the CT data using the current matching result, and comparing the corresponding pairs of the calculated pairs to invalid matching. A matching result determination step of determining whether to match, a matching pair removing step of removing the corresponding pair if it is an invalid match, and a matching result calculating step of calculating a matching result of the current step using the valid matching pair; Is repeatedly performed until the repetition termination condition is not satisfied. The repetition termination condition is characterized in that there are three or less valid pairs, a predetermined number of repetitions, or a conversion matrix converges.

In addition, the determination of whether the match is invalid may be performed by comparing Measure 2, a characteristic representing curvature determined by the geometric shape of the point set around the corresponding pair of each corresponding pair, and further comparing the distance between vertices of each corresponding pair. It is characterized by determining whether or not an invalid match.

In addition, the improved matching performance of the CT data and the optical data according to an embodiment of the present invention, the initial matching unit for performing the initial matching by applying a plurality of corresponding points to the CT data and the optical data, CT performing the initial matching A surface vertex extracting unit extracting surface vertex candidates of data and optical data, calculating an error measure of the extracted surface vertex candidates, and calculating the calculated error value for each CT data and surface vertex candidates of the optical data A vertex sampling unit that removes invalid vertices by performing vertex sampling, a matching result calculating unit calculating a matching result based on the sampling result, and reconstructing a three-dimensional model by performing matching based on the calculated final matching result And a matching processing unit.

The initial matching unit may perform initial matching by applying three pairs of corresponding points to the CT data and the optical data.

In addition, the vertex sampling unit, in the case of the CT data, in the case of the CT data, represents a curvature determined according to Measure 1 calculated based on a CT value and a geometric shape of a point set around a corresponding position. Noise is removed by using Measure 2, and the noise is determined to have a large change in ambient intensity at each location using Measure 1 and a large change in curvature at each location using Measure 2 as noise. In the case of the optical data, it is determined that the change in the curvature of the surroundings at each position is large as noise using Measure 2 to remove the noise.

The matching result calculating unit may calculate a corresponding point of the optical data surface vertex with respect to the surface vertex candidate of the CT data by using the current matching result, compare each of the calculated pairs, and determine whether the matching is invalid. If the match is not valid, the method further includes removing the corresponding pair and calculating a matching result of the current step by using the valid matching pair, and performing the repetition until the repetition termination condition is not satisfied. Or 3 or less valid pairs, a predetermined number of repetitions, or a convergence matrix.

In addition, the matching result calculating unit compares Measure 2, which is a characteristic indicating curvature determined according to the geometric shape of the point set around the corresponding position of each pair when determining whether the match is invalid, and further corresponds to each correspondence. It is characterized by comparing the distance between the vertices of the pair to determine whether an invalid match.

As described above, according to the method and apparatus for improving the matching performance of the CT data and the optical data of the present invention, when reconstructing the internal and external structures of an object in three dimensions, only some highly reliable vertices are used without using all surface vertex information. By matching the data with the optical data, the time required for matching can be greatly shortened, the complexity of the matching process can be reduced, and the accuracy of the matching result can be improved.

In addition, the present invention improves the matching performance of the CT data and the optical data, thereby deriving surface information including the inhomogeneous noise generated in the past, calculating vertices that do not need to be considered, and converging due to outliers. There is an effect that can solve the interruption.

In addition, the present invention has the effect of performing a lesion diagnosis, pre-operative simulation, a surgical guide using augmented reality using the internal and external information of the object through a three-dimensional model configured with high accuracy within a short time.

1 is a view for explaining a method that can determine the vertices of high reliability in the CT data and optical data proposed in the present invention.

2 is a view for explaining an example of a noise removal process using Measure 1 (Uncertainty) proposed in the present invention.

3 is a view for explaining an example of a noise removal process using Measure 2 (Curvature) proposed in the present invention.

4 is a diagram schematically illustrating a configuration of an apparatus for improving matching performance of CT data and optical data according to an embodiment of the present invention.

FIG. 5 is a diagram showing the configuration of the matching performance improving apparatus of FIG. 4 in more detail.

6 is a flowchart illustrating an operation process of a method of improving matching performance of CT data and optical data according to an embodiment of the present invention.

7 is a flowchart illustrating an operation process of calculating the matching result of FIG. 6 in more detail.

8 is a diagram for describing performing a plurality of corresponding point inputs for initial matching.

9 is a diagram illustrating an example of an initial optical data surface candidate.

FIG. 10 illustrates an example of an optical data surface candidate after noise is removed by using Measure 2 (Curvature) proposed in the present invention.

11 is a diagram illustrating an example of an optical data surface candidate used for calculating an actual matching result.

12 is a diagram illustrating an example of a final matching result.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the method and apparatus for improving the matching performance of the CT data and optical data of the present invention. Like reference numerals in the drawings denote like elements. In addition, specific structural to functional descriptions of the embodiments of the present invention are only illustrated for the purpose of describing the embodiments according to the present invention, and unless otherwise defined, all terms used herein including technical or scientific terms These have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and are not construed in ideal or excessively formal meanings unless expressly defined herein. It is preferable not to.

First, prior to the detailed configuration description of the present invention, the necessity of the present invention is applied to a method of determining a vertex with high reliability rather than using all the vertex information extracted as in the conventional matching method.

1 is a view for explaining a method that can determine the high reliability in the CT data and optical data proposed in the present invention, Figures 2 and 3 are Measure 1 (Uncertainty) and Measure 2 proposed in the present invention A diagram for describing an example of a noise removing process using (Curvature).

In the present invention, in order to extract reliable vertices from the CT data and the optical data, Measure 1 is calculated based on the CT value (intensity) and measures representing the curvature (Curvature) determined according to the geometric shape of the point set around the position Two ways are suggested.

In addition, in the present invention, Measure 1 and Measure 2 are used to determine reliable vertices in CT data, and Measure 2 is used to determine reliable vertices in optical data.

FIG. 1 is a view for explaining a method for determining reliable vertices in CT data and optical data proposed in the present invention, and FIG. 1 (a) shows CT values among point sets extracted from CT data. Figure 1 is a measure of the probability of noise using Measure 1 calculated based on Figure 1 (b) and (c) is the geometric of the point set around the corresponding position among the point set or mesh point set extracted from the CT data It is a figure measuring the possibility of noise using Measure 2 which shows the curvature determined according to shape. Highly likely noise means low reliability.

In this case, Measure 1 is a value calculated based on the CT value (intensity) and can be used only in the CT data. In general, CT data is data having a form in which an intensity is filled in a space divided by a grid. to be. In such CT data, noise is generated due to the physical problem mentioned in the conventional problem, and this noise often occurs suddenly larger in intensity than the surrounding values. That is, Measure 1 represents a large value when a change in intensity is large at each position, and a point having a high probability of noise may be distinguished using the measured value.

2 is a view for explaining an example of a noise removing process using the Measure 1 (Uncertainty) proposed in the present invention, a point set extracted as iso-value from CT data (the stronger the green the higher the uncertainty (Uncertainty) Figure 1 shows the iso-points after removing points with high uncertainty (Uncertainty) using Measure 1.

In addition, Measure 2 is a characteristic indicating curvature determined according to the geometric shape of a point set (iso-Point extracted from CT data, a point forming a mesh) around a corresponding position. That is, the degree of bending of the surface around the location is shown. Therefore, Measure 2 may be used as one of criteria for determining whether two points are geometrically similar in shape when determining a correspondence between an input mesh point and an iso-point extracted from CT. In addition, when Measure2 appears very large in a narrow region, it may be determined that the corresponding position is likely to be noise.

3 is a view for explaining an example of a noise removal process using the Measure 2 (Curvature) proposed in the present invention, in the set of points of the input mesh (the stronger the green the higher the Measure 2 (Curvature) value) Measure 2 shows the set of input mesh points after removing noise.

That is, in the present invention, the noise is removed using Measure 1 calculated based on the CT value (intensity) and Measure 2 representing the curvature (Curvature) determined according to the geometric shape of the point set around the position in the case of CT data, In the case of optical data, noise is removed using Measure 2, thereby shortening the accuracy and time of the result.

As shown in FIG. 4, the present invention includes a matching performance enhancing apparatus 100, an image photographing apparatus 200, a database 300, and a display apparatus 400.

In more detail with respect to the matching performance improvement process of the present invention configured as described above, the CT image of the object to be photographed in the image recording apparatus 200 is transmitted to the matching performance improving apparatus 100 (①), At the same time, the optical data of the object to be photographed is photographed and transmitted to the matching performance improving apparatus 100 (②). The matching performance enhancing apparatus 100 extracts the surface vertices of the CT data and the optical data input from the image photographing apparatus 200 based on the proposed method (③), and then uses the extracted vertices to generate the CT data and the optical. Data matching is performed (④), the 3D model is reconstructed based on the matching result, and stored in the database 300 (⑤), and the display apparatus 400 stores the 3D model which is the final matching result. Display it through the user for confirmation (⑥).

The matching performance enhancing apparatus 100 has an advantage of CT data that can reconstruct an internal and external structure in three dimensions when the CT data photographed by the image photographing apparatus 200 and the optical data are matched using an ICP algorithm. It is to improve the matching performance of CT data and optical data by taking advantage of high precision optical data with 3D surface to be reconstructed evenly, and only some highly reliable vertices do not need to use all surface vertex information as conventionally. Because the matching is performed, the time can be reduced while increasing the accuracy of the reconstruction results of the 3D model.

That is, the matching performance enhancing apparatus 100 performs initial matching by applying a plurality of corresponding points to the CT data and the optical data received from the image photographing apparatus 200, and performs surface matching from the CT data and the optical data on which the initial matching is performed. Extract the vertex candidates. After sampling the surface vertex candidates of the CT data and the optical data to remove the invalid vertices, the CT data and the optical data are matched only based on the valid vertex information, and based on the matching result, The reconstruction is performed and stored in the database 300 or in a storage device provided by itself, and displayed through the display device 400 when necessary.

Accordingly, the matching performance enhancing apparatus 100 derives surface information including heterogeneous noise generated in the conventional matching process, calculates vertices that do not need to be considered, and prevents convergence due to outliers. The problem can be solved. As a result, the time required for matching is greatly reduced, the complexity of the matching process is reduced, and the accuracy of the matching result can be improved.

In addition, the matching performance enhancing apparatus 100 may improve the matching performance of CT data and optical data to construct a three-dimensional model with high accuracy in a short time, so that lesion diagnosis, preoperative simulation, and augmentation using internal and external information of an object may be performed. Surgery guide using the reality can be performed efficiently.

The image capturing apparatus 200 collectively refers to equipment such as a known CT camera, scanner, etc., is connected to the matching performance enhancing apparatus 100, and is obtained by projecting an object to be photographed by X-ray from various angles. The CT data is provided to the matching performance improving device 100, and the optical data of a photographing target object photographed by a scanner or the like is provided to the matching performance improving device 100.

The database 300 stores and manages CT data and optical data photographed for each object provided by the image capturing apparatus 200 in the matching performance improving apparatus 100, as well as the matching performance improving apparatus 100. We store and manage the reconstructed 3D model according to the matching result in.

In addition, the database 300 stores and updates various operation programs for improving the matching performance used in the matching performance improving apparatus 100.

The display device 400 is a monitor such as a conventional LCD, LED, etc., and displays a three-dimensional model reconstructed according to the matching result of the CT data and the optical data in the matching performance improving apparatus 100 on a screen so that the user Make sure to check.

5 is a diagram illustrating the configuration of the matching performance improving apparatus 100 of FIG. 4 in more detail.

As shown in FIG. 5, the matching performance improving apparatus 100 includes a CT data input unit 110, an optical data input unit 120, an initial matching unit 130, a surface vertex extracting unit 140, and a vertex sampling unit ( 150, a matching result calculator 160, a matching processor 170, a storage unit 180, a controller 190, and the like.

In addition, although not shown in the drawing, the matching performance improving apparatus 100 may further include a power supply unit for supplying operation power to each component, an input unit for performing key signal input for setting various functions, and the like.

The CT data input unit 110 receives CT data of a specific object photographed by the CT imager from the image capturing apparatus 200 and transmits the CT data to the controller 190.

The optical data input unit 120 receives optical data of a specific object photographed using the scanner in the image capturing apparatus 200 and transmits the optical data to the controller 190.

In addition, each of the CT data input unit 110 and the optical data input unit 120 converts CT data and optical data provided from the image photographing apparatus 200 into data for use in the matching performance enhancing apparatus 100. Perform pretreatment.

The initial matching unit 130 performs initial matching by applying a plurality of corresponding points to the CT data provided from the CT data input unit 110 and the optical data provided from the optical data input unit 120.

Here, the initial matching unit 130 has been described as an example of performing initial matching by applying three pairs of corresponding points to the CT data and the optical data, but the present invention is not limited thereto. Note that it can be applied.

The surface vertex extracting unit 140 extracts the surface vertex candidates of the CT data and the optical data which have been initially matched through the initial matching unit 130. For example, in the case of the CT data, surface vertex candidates are extracted based on an iso-surface based on CT values.

The vertex sampling unit 150 calculates an error measure of the surface vertex candidate extracted by the surface vertex extracting unit 140, and calculates the calculated error value for the surface vertex candidates of the CT data and the optical data. Based on the vertex sampling, we remove the invalid vertices.

The vertices extracted from the CT data have Measure 1 and Measure 2 calculated from the CT data, and the vertices extracted from the optical data have Measure 2 calculated based on the geometric distribution of the vertices.

In more detail, the vertex sampling unit 150 calculates curvature determined according to Measure 1 calculated based on the CT value and the geometric shape of the point set around the position in the case of the CT data. Remove the noise using Measure 2, which is shown. That is, noise 1 determines that the change in the surrounding value (ie intensity) is large at each position using Measure 1 and the change in the curvature of the surrounding at each position is large using Noise Measure 2 is removed.

In addition, unlike the CT data, the vertex sampling unit 150 determines that the change in the curvature of the surroundings is large as noise by using only Measure 2, unlike the CT data. This is because Measure 1 is calculated based on intensity and can only be calculated in CT data.

The matching result calculator 160 calculates a matching result based on the sampling result performed by the vertex sampling unit 150 and provides the matching result to the controller 190.

More specifically, the matching result calculating unit 160 calculates the corresponding points of the optical data surface vertices with respect to the surface vertex candidates of the CT data using the current matching results, and then compares the corresponding pairs. To determine if there is an invalid match. If it is determined that the match is invalid, the corresponding pair is removed, and then the matching result of the current step is calculated using the valid pair. The calculation of the matching result is performed repeatedly until the repeat termination condition is not satisfied. In this case, the repetition end condition is that when there are three or less valid pairs (three minimum valid pairs for calculating the transformation matrix (ie, matching result)), and when the predetermined number of iterations is reached, the transformation matrix converges (that is, before This means that the repetition order and the result are the same).

In this case, when the matching result calculating unit 160 determines whether the match is invalid, a measure representing a curvature determined according to a geometric shape of a point set around a corresponding position of each corresponding pair of CT data and optical data is measured. Compare 2 and further compare the distances between the vertices of each pair (ie, determine that they are not valid if the distance between the pairs of pairs is too large) to determine whether they match.

The matching processor 170 reconstructs the 3D model by performing matching based on the final matching result calculated by the matching result calculator 160.

The storage unit 180 stores various operation programs used in the matching performance improving apparatus 100, and updates the respective operation programs through the database 300.

In addition, the storage unit 180 stores CT data and optical data photographed by the image capturing apparatus 200, and stores a 3D model reconstructed by the matching processing unit 170.

The control unit 190 is a part that collectively controls the operation of the matching performance improving device 100, CT data input and pre-processing in the CT data input unit 110, optical data input in the optical data input unit 120 And pre-processing, and storing the storage unit 170 of the CT data and optical data.

In addition, the control unit 190 is the initial matching of the CT data and the optical data by applying a plurality of corresponding points in the initial matching unit 130, the initial matching of the CT data and the optical data in the surface vertex extraction unit 140 Extracting vertex candidates, performing vertex sampling on surface vertex candidates of respective CT data and optical data in the vertex sampling unit 150, and controlling the elimination of invalid vertices.

In addition, the controller 190 calculates a final matching result using the vertex sampling result in the matching result calculating unit 160, reconstructs and reconstructs a 3D model matched based on the final matching result in the matching processing unit 170. Controls the screen display through the display device 400 of the storage unit 180 or the database 300 for the stored and reconstructed 3D model.

Next, an embodiment of a method for improving matching performance of CT data and optical data according to the present invention configured as described above will be described in detail with reference to FIGS. 6 to 12. At this time, each step according to the method of the present invention can be changed in order by the environment or those skilled in the art.

6 and 7 are flowcharts showing in detail the operation of the method for improving the matching performance of the CT data and the optical data according to an embodiment of the present invention, Figures 8 to 12 are processing results according to the performance of each step of the present invention One example of each drawing.

As shown in FIG. 6, the matching performance enhancing apparatus 100 receives CT data photographed by a CT imager and optical data photographed by a scanner from the image photographing apparatus 200 with respect to the same object.

The matching performance enhancing apparatus 100 performs initial matching by applying a plurality of corresponding points to the CT data and the optical data received from the image photographing apparatus 200 (S100). In this case, it is preferable that the matching performance improving apparatus 100 performs initial matching by applying three pairs of corresponding points to the CT data and the optical data. For example, as shown in FIGS. 8A and 8B, three corresponding points required for initial matching are applied.

After performing the initial matching through the step S100, the matching performance enhancing apparatus 100 extracts the surface vertex candidates of the CT data and the optical data that performed the initial matching (S200). For example, FIG. 9 is an example of a surface vertex candidate of initial optical data.

After extracting the surface vertex candidate through the step S200, the matching performance enhancing apparatus 100 calculates an error measure of the extracted surface vertex candidate (S300). That is, in the case of the CT data, Measure 1, which is calculated based on the CT value (intensity), and Measure 2 which represents the curvature (Curvature) determined according to the geometric shape of the point set around the corresponding position, are identified.

Thereafter, the matching performance enhancing apparatus 100 performs vertex sampling based on the error value calculated in step S300 on the respective CT data and the surface vertex candidates of the optical data extracted in step S200 to remove invalid vertices. (S400). That is, in the case of CT data, noise is determined by removing a large change in the surrounding value (ie intensity) at each position using Measure 1 and a large change in the curvature at each position using Measure 2 as noise. In the case of using only Measure 2, it is determined that the change in the curvature of the surroundings at each position is large as noise and removed. For example, FIG. 10 is an optical data surface vertex candidate after noise is removed using Measure 2, and FIG. 11 is an optical data surface vertex candidate used for calculating the actual matching result to be performed in the next step S500.

Now, after removing the invalid vertex through the vertex sampling of the surface vertex candidate through the step S400, the matching performance enhancing apparatus 100 calculates a matching result based on the sampling result (S500).

Referring to FIG. 7 in more detail with reference to FIG. 7, the matching performance enhancing apparatus 100 calculates a corresponding point of the optical data surface vertex with respect to the surface vertex candidate of the CT data using the current matching result (S510). In operation S520, the corresponding surface pairing candidate is compared to determine whether the current candidate for surface correction is not valid (S520).

In this case, the determination of whether the match is invalid in operation S530 may be performed by comparing Measure 2, which is a characteristic representing curvature determined according to the geometric shape of the point set around the corresponding position of each pair, and further vertices of each pair. By comparing the distance between the two to determine whether the match is invalid.

If the determination result of the step S530 is that the current surface correction candidate is invalid, the matching performance improving apparatus 100 removes the corresponding pair (S540), and calculates the matching result of the current step using the valid pair. (S550).

The matching performance enhancing apparatus 100 determines whether there is a surface vertex candidate for calculating the matching result (S560), and repeats the process after step S510 until there is no vertex candidate for the surface for calculating the matching result. To do it.

The matching performance improving apparatus 100 determines whether the repetition termination condition is satisfied (for example, when there are three or less valid pairs, a predetermined number of repetitions, or a transformation matrix converges). S560), and after the step S510 is repeatedly performed until the repetition end condition is not satisfied.

When the final matching result is calculated through the step S500, the matching performance enhancing apparatus 100 reconstructs the 3D model by performing matching based on the final matching result (S600). For example, FIG. 12 is a diagram illustrating a result of matching performed based on the final matching result.

The 3D model reconstructed through the step S600 may be stored in the database 300 or in a storage device provided by itself, and displayed on a screen through the display device 400 so that the user can check it.

As described above, the present invention has been described with reference to the embodiment shown in the drawings, but this is merely exemplary, and those skilled in the art to which the art belongs, various modifications and equivalent other embodiments therefrom I understand that it is possible. Therefore, the technical protection scope of the present invention will be determined by the claims below.

In the present invention, since the matching of CT data and optical data is performed using only some reliable vertices without using all surface vertex information when reconstructing the internal and external structures of an object in three dimensions, the time required for matching is reduced, and the matching is performed. The complexity of the process can be reduced and the accuracy of matching results can be increased.

In addition, surface information including non-homogeneous noise generated in the related art can be derived, computed to vertices that do not need to be considered, and solve the problem of preventing convergence due to outliers.

In addition, lesion diagnosis, preoperative simulation, and surgical guide using augmented reality can be performed by using a three-dimensional model configured with high accuracy within a short time.

Claims

An initial matching step of performing an initial matching by applying a plurality of corresponding points to the CT data and the optical data in the matching performance improving apparatus;

A surface vertex candidate extraction step of extracting surface vertex candidates of the CT data and the optical data on which the initial matching is performed;

Calculate an error measure of the extracted surface vertex candidates, and perform vertex sampling on the surface vertex candidates of the CT data and the optical data based on the calculated error value to remove invalid vertices. step;

A matching result calculating step of calculating a matching result based on the sampling result; And

And a matching step of reconstructing a 3D model by performing matching based on the calculated final matching result.
The method according to claim 1,

The initial matching step,

And performing initial matching by applying three pairs of corresponding points to the CT data and the optical data.
The method according to claim 1,

The vertex sampling step,

In the case of the CT data, noise is removed using Measure 1, which is calculated based on the CT value, and Measure 2, which represents a curvature determined according to the geometric shape of the point set around the corresponding position. By using 1, the change in the intensity of the surroundings at each position is large and the change in the curvature of the surroundings at each position is large using the Measure 2 is determined as noise and removed.

In the case of the optical data, the measure 2 improves the coherence performance of the CT data and the optical data, by using the Measure 2 to determine that noise has a large change in the surrounding curvature at each position.
The method according to claim 1,

The matching result calculating step,

A correspondence point calculation step of calculating a correspondence point of the optical data surface vertex with respect to the surface vertex candidate of the CT data using the current matching result;

A matching determination step of determining whether or not an invalid match is made by comparing the calculated pairs;

A corresponding pair removing step of removing the corresponding pair if it is an invalid match; And

And a matching result calculating step of calculating a matching result of the current step using a valid pair.

Each step is performed repeatedly until the repeat end condition is not satisfied.

The repetition termination condition is a method of improving the matching performance of CT data and optical data, characterized in that the number of valid pairs is three or less, a predetermined number of repetitions is reached, or a conversion matrix is converged.
The method according to claim 4,

The determination of whether the match is invalid,

Compare Measure 2, a characteristic representing curvature determined by the geometric shape of a set of points around a corresponding position of each pair, and further compare distances between vertices of each pair to determine invalid matching A method of improving the matching performance between CT data and optical data.
An initial matching unit configured to perform initial matching by applying a plurality of corresponding points to the CT data and the optical data;

A surface vertex extracting unit extracting surface vertex candidates of the CT data and the optical data on which the initial matching is performed;

Calculate an error measure of the extracted surface vertex candidates, and perform vertex sampling on the surface vertex candidates of the CT data and the optical data based on the calculated error value to remove invalid vertices. part;

A matching result calculator for calculating a matching result based on the sampling result; And

And a matching processor configured to reconstruct a three-dimensional model by performing matching based on the calculated final matching result. 2.
The method according to claim 6,

The initial matching portion,

And an initial matching operation by applying three pairs of corresponding points to the CT data and the optical data.
The method according to claim 6,

The vertex sampling unit,

In the case of the CT data, noise is removed using Measure 1, which is calculated based on the CT value, and Measure 2, which represents a curvature determined according to the geometric shape of the point set around the corresponding position. By using 1, the change in the intensity of the surroundings at each position is large and the change in the curvature of the surroundings at each position is large using the Measure 2 is determined as noise and removed.

In the case of the optical data, by using Measure 2, it is determined that the change in the curvature of the surroundings is large at each position as noise to remove the matching performance of the CT data and the optical data.
The method according to claim 6,

The matching result calculation unit,

Compute the corresponding points of the optical data surface vertices with respect to the surface vertex candidates in the CT data using the current matching results, compare each of the calculated pairs, determine whether an invalid match is found, and if it is an invalid match, Removing the pair, and calculating the matching result of the current step using the valid corresponding pair,

Iteratively executes until the iteration termination condition is not satisfied.

The repetition termination condition is a case where there are three or more valid corresponding pairs, a predetermined number of repetitions is reached, or a conversion matrix is converged.
The method according to claim 9,

The matching result calculation unit,

In determining whether the match is invalid, Measure 2, a characteristic representing curvature determined according to the geometric shape of a set of points around a corresponding position of each pair of pairs, is further compared, and the distance between vertices of each pair is further compared. And determining an invalid match to determine match performance of CT data and optical data.