KR20230089889A - Medical image matching device for enhancing augment reality precision of endoscope and reducing deep target error and method of the same - Google Patents

Abstract

To enhance the augmented reality precision of endoscopy and reducing deep target lesion error, a medical image matching device according to an embodiment of the present invention comprises: an input part for receiving the image data of a patient space and medical image data; an artificial intelligence model storage part for storing an artificial intelligence model for estimating the rotation and translation transformation matrix values for precise matching based on the image data of the patient space and the medical image data; a matching part for precisely matching the image data of the patient space and the medical image data; and an output part for outputting the matching result of the image data of the patient space and the medical image data, wherein the matching part may include: a first matching part for generating a first matching coordinate point by roughly matching the image data of the patient space and the medical image data, a second matching part for generating a second matching coordinate point by matching the first matching coordinate point and a surface coordinate point of the medical image data; and a third matching part for performing precise matching based on the rotation and translation transformation matrix values estimated using the second matching coordinate point as an input into the artificial intelligence model.

Description

Medical image matching device and method for enhancing endoscopic augmented reality precision and reducing error in deep lesions

The present invention relates to a medical image matching device, and more particularly, to a medical image matching device capable of precisely matching medical image data and patient space data by improving the precision of augmented reality of an endoscope and minimizing errors in deep lesions. and its method.

More specifically, the present invention uses an artificial neural network to estimate rotational and translational transformation matrix values based on patient space data and phantom lesion position information, and precisely match medical image data data and patient space data. It relates to a medical image matching device capable of performing and a method therefor.

An image-guided surgical system is a medical system that visualizes the three-dimensional position of a surgical instrument on a pre-surgery medical image of a patient and provides the position of a lesion and its surroundings in real time.

The image-guided surgical system acquires a medical image of a patient before surgery, performs spatial registration during surgery, and displays a virtual surgical tool overlaid on the medical image through coordinate conversion of the surgical tool. Here, spatial matching refers to a process of aligning the coordinates of a medical image taken before an operation with the coordinates of a patient's space obtained during an operation in the same space. If the two coordinates are not accurately matched, a gap may occur between the position of the actual surgical tool and the position of the virtual surgical tool. Therefore, a technique for accurately matching the prepared medical image and the image of the patient's space is required.

Meanwhile, such spatial matching is divided into point matching and surface matching. First, point matching acquires coordinates of at least three marker markers that do not exist on a single straight line in the space of a medical image and a patient. In addition, rotation and linear transformation are performed between coordinate systems so that the alignment of the two spaces can be matched based on the local coordinate system of each medical image and patient space.

Therefore, in order to perform point matching, a marker marker must be attached to the patient's face in advance, and additionally, computed tomography (CT) imaging is required. In addition, there is a high possibility of swelling of the body part due to the marker marker attached to the patient's face, and due to this, there is a limitation that accuracy is lowered when the marker marker moves.

On the other hand, Figure 1 is a view for explaining the conventional surface matching.

As shown in FIG. 1 , information on the surface of the patient P may be acquired through the probe 10 .

For example, the surgeon moves the probe 10 slowly while the probe 10 is adjacent to the body of the patient P, and the surgical navigation device provides probe coordinate data reflecting information on the surface of the patient P (C) will be obtained.

In addition, the surgical navigation device rotates or moves the acquired probe coordinate data (C) to match the surface coordinate data obtained from the previously photographed medical image of the patient (P) and to position the matched result in a single space do.

Meanwhile, since the surface coordinate data is based on high-resolution image information acquired through medical imaging equipment, it may consist of thousands to hundreds of thousands of coordinate data.

Therefore, surface matching finds a transformation matrix as a set of coordinates in each space without a designated marker marker between the patient space and the medical image. In addition, surface matching does not require an additional CT scan and is convenient for clinicians.

However, surface matching causes a problem of increased lesion error as the affected area moves away from the facial surface, which is an area where a patient spatial coordinate cluster is acquired. For example, when a slight rotation error occurs on the face surface, it is amplified on the side of the deep lesion far from the face surface. That is, since surface matching is performed only with the coordinates of the face surface, there is a limit in that an error may occur on the side of a deep lesion and may be mistaken for another lesion.

A technical problem to be achieved by the present invention relates to a medical image matching device, and more particularly, to estimate rotational and translational transformation matrix values based on patient space data and lesion position information of a phantom, and deep lesion errors. An object of the present invention is to provide a medical image matching device capable of precisely matching data of medical image data and data of a patient's space by minimizing the size.

The technical problem to be achieved by the present invention is not limited to the above-mentioned technical problem, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

In order to achieve the above technical problem, in the medical image matching device for precisely matching image data of a patient space and medical image data in order to improve precision of endoscopic augmented reality and reduce errors in deep lesions according to an embodiment of the present invention, the patient An input unit for receiving spatial image data and the medical image data, and an artificial intelligence model storing an artificial intelligence model for estimating rotational and translational transformation matrix values for precise matching based on the image data of the patient's space and the medical image data. It may include a matching unit that precisely matches the patient space image data and the medical image data, and an output unit that outputs a matching result of the patient space image data and the medical image data.

In addition, the matching unit generates a first matching coordinate point by roughly matching the image data of the patient space and the medical image data, and the first matching coordinate point and the surface coordinate point of the medical image data. A second matching unit generating a second matching coordinate point by matching and a third matching unit performing precise matching based on the rotation and translation matrix values estimated by using the second matching coordinate point as an input of the artificial intelligence model can do.

The input information feature points of the artificial intelligence model include face surface coordinate points in the patient space and surface coordinate points of the medical image data corresponding to the face surface coordinate points in the patient space, and are set in six dimensions. can

The artificial intelligence model further secures a data set based on the coordinates of the lesion location of the phantom to set input feature points of the artificial intelligence model, and the input feature points of the artificial intelligence model based on the coordinates of the lesion location are the virtual lesion structures. It includes the location coordinate points of the lesion candidate group arbitrarily designated in , and the location coordinate point corresponding to the location information of the lesion candidate group in the image data obtained by photographing the lesion structure, and can be set in six dimensions.

The artificial intelligence model includes a Bi-LSTM (Bidirectional LSTM) architecture, and a rotation and translation matrix can be output as a 3X3-dimensional rotation matrix and a 3X1-dimensional translation matrix using a regression layer. .

The second matching unit, the artificial intelligence model, and the third matching unit may be matched using an iterative closest point (ICP) algorithm.

In addition, in the medical image precision matching method for precisely matching patient space image data and medical image data in order to improve endoscopic augmented reality precision and reduce deep lesion errors according to an embodiment of the present invention, a) the image of the patient space Acquiring data and the medical image data; b) roughly extracting and matching coordinate points from the patient space image data and the medical image data, and extracting a rough rotation and translation matrix; c) First matching the roughly matched coordinate points and facial surface coordinate points extracted from the image data of the patient space using the extracted coarse rotation and translation matrix, and generating a first matching coordinate point, d) generating a second matching coordinate point by augmenting the first matching data and matching the augmented first matching coordinate point with a surface coordinate point extracted from the medical image data; and and performing precise matching based on the rotation and translation matrix values estimated as inputs of the artificial intelligence model.

Step d) and step e) may be matched using an iterative closest point (ICP) algorithm.

In the step d), the first matching coordinate point is augmented by 5% for each augmentation, and the augmentation may be performed a plurality of times.

The step d) includes analyzing a surface matching error every time the one-time augmentation is finished, and the augmentation may be stopped when the surface matching error analysis value satisfies a preset end condition.

According to an embodiment of the present invention, since the image data of the patient space and the medical image data are matched using a rotation and translation matrix extracted based on an artificial intelligence model, errors in deep lesions occurring during surface matching are minimized. can do.

In addition, according to an embodiment of the present invention, matching accuracy may be improved by first performing course matching by roughly extracting coordinate points from image data of the patient space and medical image data.

In addition, according to an embodiment of the present invention, since a data set including coordinate points of the lesion location of a virtual lesion structure is obtained, and learning is performed by setting feature points of artificial intelligence model input information based on this, learning is performed by Optimized rotational and translational transformation matrices can also be estimated on the deep lesion side.

In addition, according to an embodiment of the present invention, the user can recognize a result of matching medical image data and patient space image data in which an error on the side of a deep lesion is minimized. As a result, the user can accurately grasp the deep lesion and can efficiently perform surgery on the deep lesion.

The effects of the present invention are not limited to the above effects, and should be understood to include all effects that can be inferred from the detailed description of the present invention or the configuration of the invention described in the claims.

1 is a diagram for explaining conventional surface matching.
2 is a diagram illustrating a medical image matching device according to an embodiment of the present invention.
3 is a diagram illustrating a method of matching surface data of medical image data with face surface data of a patient's space according to an embodiment of the present invention.
4 is a diagram specifically illustrating a method of performing first and second registration by a medical image registration apparatus according to an embodiment of the present invention.
5 is a diagram illustrating a method of building an artificial intelligence model according to an embodiment of the present invention.
6 is a diagram illustrating input information for learning an artificial intelligence model according to an embodiment of the present invention.
7 is a diagram showing the Bi-LSTM architecture of an artificial intelligence model according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and, therefore, is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected (connected, contacted, combined)" with another part, this is not only "directly connected", but also "indirectly connected" with another member in between. "Including cases where In addition, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a diagram illustrating a medical image matching device according to an embodiment of the present invention.

As shown in FIG. 2, the medical image matching device 100 according to an embodiment of the present invention includes an input unit 110, a coordinate point generation unit 120, an augmentation unit 130, and an artificial intelligence model storage unit 140. , may include a matching unit 150 and an output unit 160.

First, the input unit 110 receives image data of a patient's space and medical image data. The patient space is a space in which a patient is located, and may be obtained from the patient space during surgery. Also, the image data of the patient's space and the medical image data may be in a 3D form.

For example, the input unit 110 may include a probe (not shown). The probe may acquire image data of the patient's space. The probe may be a rod-shaped medical tool, and may be used with an end disposed in contact with or adjacent to a patient's body surface. The probe moves along the patient's body surface and acquires image data of the body surface in the patient's space. The probe may include a sensor disposed on one side facing the patient's body. The sensor may be an optical sensor.

Meanwhile, the medical image data is an image of a patient acquired before surgery and may be captured through CT, MRI, or ultrasound.

Also, the coordinate point generator 121 may include a first coordinate point generator 121 and a second coordinate point generator 122 .

The first coordinate point generation unit 121 may obtain a set of surface coordinate points from medical image data. Also, the first coordinate point generation unit 121 may generate a first coordinate point, which is a reference coordinate point for coarse matching, based on the acquired surface coordinate point set. Here, the first coordinate point is a surface coordinate point of medical image data extracted for approximate matching.

The second coordinate point generation unit 122 may obtain the coordinate points of the patient's body surface input from the probe and generate second coordinate points that are reference coordinate points for course matching. Here, the second coordinate point may include a coordinate point obtained when the probe contacts the surface of the patient's face. For example, the second coordinate point may include a face surface coordinate point in the patient's space when the probe contacts the forehead, nose tip, eye tip, and forehead of the patient.

The augmentation unit 130 augments the first matched surface coordinate points. As will be described later, the augmentation unit 130 may augment the first matching coordinate point using an interpolation method. Here, augmenting the first matching coordinate points may be understood as increasing the number of first matching coordinate points. Specifically, the augmentation unit 130 may augment the first matching coordinate point using a piecewise cubic Hermite interpolating polynomial. The piecewise cubic Hermitian interpolation polynomial has a feature of interpolating by matching the value of each point with the first derivative. The piecewise cubic Hermitian interpolation polynomial can be described by the following [Equation 1].

[Equation 1]

[Equation 2]

, 데이터 값을

, 기울기 값을

, k번째 데이터의 구간을

,

라 하고,

의 범위에서 [수학식 2]를 만족하는 [수학식 1]의 3차 다항식

를 계산하는 것을 의미한다.The piecewise cubic Hermite interpolation method takes the kth data

, the data value

, the slope value

, the interval of the kth data

,

say,

A cubic polynomial of [Equation 1] that satisfies [Equation 2] in the range of

means to calculate

If coordinate points are augmented using piecewise cubic Hermitian interpolation, an overshoot phenomenon in which data rises excessively can be prevented. Therefore, it is possible to precisely connect the flat areas of the body surface.

Also, the augmentation unit 130 may augment the first matching coordinate points by 5%. Assuming that the process of augmenting the first registration coordinate point by 5% is one step, the augmentation unit 130 may repeat one or more steps. The augmentation unit 130 augments the number of coordinate points of the first matching coordinate points acquired within a limited period of time, thereby increasing matching accuracy.

The artificial intelligence model storage unit 140 may receive medical image data and patient space image data, and estimate rotation and translation matrix values using the received medical image data and patient space image data.

Specifically, an artificial intelligence model may be stored in the artificial intelligence model storage unit 140, and the artificial intelligence model may estimate rotation and translation matrix values for precision matching by taking medical image data and patient space image data as inputs. can The artificial intelligence model can be implemented as an artificial intelligence model LSTM (Long Short Term Memory) that learns characteristic values by itself over time and predicts them according to correlated time-series data. In addition, the artificial intelligence model may include Bidirectional LSTM (Bi-LSTM), which uses LSTM together with forward as well as backward results in order to improve accuracy.

Meanwhile, the matching unit 150 may include a first matching unit 151 , a second matching unit 152 , and a third matching unit 153 .

The first matching unit 151 may roughly match the first coordinate point and the second coordinate point. Registration means arranging sets of coordinate points located in their respective spaces in one coordinate system. That is, the first matching unit 151 may arrange the matched result in one spatial coordinate system. At this time, the first matching unit 151 may extract rotation and translation matrix values in case of course matching.

In addition, the first matching unit 151 may match the face surface coordinate points in the patient space with the coordinate points that are course matched using the extracted rotation and translation matrix values. The first matching unit 151 may move or rotate the coordinate points of the facial surface in the patient's space to implement matching with the coordinate points that are course-matched. In addition, the first matching unit 151 may move or rotate the coordinate points that have been coarsely matched to make course matching with the facial surface coordinate points in the patient's space. In this case, the coordinate points of the face surface of the patient space may include coordinate points of the body surface of the patient input according to the movement path of the probe in the input unit 110 . In addition, the first matching unit 151 may arrange the surface coordinate points, which are the matching results, as the first matching coordinate points in one spatial coordinate system.

The second matching unit 152 may match the first matching coordinate points augmented by the augmenting unit 130 with the surface coordinate points of the medical image data. For example, the second matching unit 152 may be matched using an iterative closest point (ICP) algorithm.

The ICP algorithm is a method of calculating and matching rotation and movement matrices in which the average difference in positions between coordinates in two spaces can be minimized. The ICP algorithm is defined by Equation 3 below.

[Equation 3] Roughly initially

와

는 각각 환자 공간의 얼굴 표면 좌표점 세트인 증강된 제1정합 좌표점 세트와 의료 영상의 표면 좌표점 세트를 의미한다.In Equation 3, R and t denote a rotation matrix and a translation matrix capable of minimizing the position error of the two sets of coordinate points,

and

denotes an augmented first registration coordinate point set, which is a face surface coordinate point set in the patient space, and a surface coordinate point set of the medical image, respectively.

Meanwhile, the second matching unit 152 may analyze the accuracy of the matching result. The second matching unit 152 may analyze a registration error or registration accuracy through surface registration error (SRE) analysis. The surface registration error is obtained by calculating and averaging the distance between one set of surface coordinate points and another set of surface coordinate points.

The surface matching error can be calculated by Equation 4 below.

[Equation 4]

는 i번째 의료 영상의 표면 좌표점을 의미하고

는 ICP 알고리즘을 통해 획득된 변환행렬이 적용된 i번째 환자 공간의 얼굴 표면 좌표점을 의미한다. 이 때, 환자 공간의 얼굴 표면 좌표점은 i 번째 증강된 제1정합 좌표점을 포함한다.In [Equation 4]

Means the surface coordinate point of the ith medical image and

denotes the coordinate point of the face surface in the ith patient space to which the transformation matrix obtained through the ICP algorithm is applied. At this time, the face surface coordinate point in the patient space includes the i-th augmented first matching coordinate point.

The second matching unit 152 may determine whether the calculated surface matching error satisfies the end condition. When the surface matching error satisfies the end condition, the second matching unit 152 may extract the surface coordinate points of the medical image corresponding to the facial surface coordinate points of the patient space and the finally augmented facial surface coordinate points of the patient space. there is.

In addition, when the surface matching error does not satisfy the end condition, the second matching unit 152 may increase the first matching coordinate point by repeating steps in the augmenting unit 130 .

The third matching unit 153 may perform matching based on the stored artificial intelligence model. At this time, the artificial intelligence model may additionally perform optimized matching using ICP by including the phantom lesion position coordinates of the phantom. Here, the phantom is a facial shape model, and may include a virtual lesion structure. In this case, the coordinate points of the lesion location of the phantom refer to a lesion candidate group arbitrarily assigned to the virtual lesion structure. Therefore, since ICP matching is implemented based on the translation and translation matrix values extracted by the third matching unit 153 by additionally including the phantom lesion position coordinates, error amplification of deep lesions can be reduced.

Similarly, the third matching unit 153 may arrange the results in one spatial coordinate system. For example, the third matching unit 153 may arrange coordinate points matched to the real-time medical image coordinate system.

The output unit 160 outputs the result arranged in the third matching unit 153 . The output unit 160 may display the matching result as an image through a display device.

3 is a diagram illustrating a method of matching surface data of medical image data with face surface data of a patient's space according to an embodiment of the present invention.

Referring to FIG. 3 , in step S110, the medical image matching device may receive medical image data and image data of a patient's space. The medical image matching device may extract and obtain surface coordinate points of medical image data of a patient from the medical image data. For example, the surface coordinate points of the medical image data include coordinate points of the patient's body surface.

Meanwhile, the medical image matching device may extract and obtain coordinate points of the face surface of the patient's space sensed by the probe from image data of the patient's space. The medical image matching device collects image data of a patient's space representing positional information of the probe using a probe.

Next, in step S120, the medical image matching device may extract coordinate points for course matching between the surface coordinate points of the obtained medical image data and the face surface coordinate points of the patient's medical image space. For example, the medical image matching device may extract a first coordinate point for rough matching, that is, course matching, from a surface coordinate point in the space of the medical image. Also, the apparatus for matching medical images may extract second coordinate points for course matching from the coordinate points of the facial surface in the patient's space. Here, the second coordinate point may include a face surface coordinate point in the patient's space when the probe contacts the forehead, nose tip, eye tip, and forehead of the patient.

In this case, the medical image matching device may obtain a course-matched coordinate point by course-matching the extracted first and second coordinate points. Also, the apparatus for matching medical images may extract rotational and translational transformation matrix values in case of course matching.

Also, the apparatus for matching medical images may match coordinate points of the facial surface in the patient's space with coordinate points that are course-matched using the extracted rotational and translational transformation matrix values. In this case, the face surface coordinate points in the patient space may include coordinate points of the patient's body surface input according to the movement path of the probe. In addition, the medical image matching device may arrange surface coordinate points, which are matching results, as first matching coordinate points in one spatial coordinate system.

In step S130, the medical image matching device may augment the first registration coordinate point. The step of augmenting the first registration coordinate point may be repeatedly performed a plurality of times. For example, the first matching coordinate point may be augmented by 5%. Assuming that the process of augmenting the first registration coordinate point by 5% is one step, the medical image matching device may repeat one or more steps. The medical image matching device can increase matching accuracy by augmenting the number of coordinate points of the first matching coordinate points obtained within a limited time.

Also, the medical image matching device may match the augmented first matching coordinate point with the surface coordinate point of the medical image data. For example, the medical image matching device may perform matching using an iterative closest point (ICP) algorithm. The ICP algorithm is a method of calculating and matching rotation and movement matrices in which the average difference in positions between coordinates in two spaces can be minimized.

Meanwhile, the medical image registration device may analyze registration error or registration accuracy through surface registration error analysis. The surface registration error is obtained by calculating and averaging the distance between one set of surface coordinate points and another set of surface coordinate points.

Also, the medical image matching device may determine whether the calculated surface matching error satisfies the end condition. When the surface registration error satisfies the end condition, the medical image matching device may extract the surface coordinate points of the medical image data corresponding to the facial surface coordinate points in the patient space and the finally augmented facial surface coordinate points in the patient space. The extracted result means the second matching coordinate point. In addition, when the surface registration error does not satisfy the end condition, the medical image registration apparatus may augment the first registration coordinate point by repeating the steps.

In step S140, the medical image matching device may perform matching based on an artificial intelligence model. In this case, the medical image matching device may perform optimized matching based on the rotation and translation matrix values extracted by using the second registration coordinate points and the phantom lesion location coordinate points as inputs of the artificial intelligence model. For example, the medical image matching device may perform optimized matching using ICP. Therefore, since the medical image matching apparatus implements optimized matching using rotational and translational transformation matrix values in which the phantom lesion location coordinate points are reflected, error amplification of deep lesions can be reduced.

In step S150, the medical image matching device may arrange and output the final matched result in one spatial coordinate system. For example, the medical image matching device may arrange a third matching coordinate point in the real-time medical image data coordinate system and display it as an image through a display device.

4 is a diagram specifically illustrating a method of performing first and second registration by a medical image registration apparatus according to an embodiment of the present invention.

As shown in FIG. 4 , in step S210, the medical image matching device may acquire a set of surface coordinate points, which is surface coordinate data, from medical image data. For example, the medical image data may be an image of a patient acquired before surgery and captured through CT, MRI, or ultrasound.

Also, the medical image matching device may generate first coordinate points for course matching from the acquired surface coordinate point set. That is, the medical image matching device roughly extracts coordinate points from the surface coordinate point set in order to improve accuracy.

In operation S220, the medical image matching device may extract face surface data from image data of the patient's space. In detail, the medical image matching device may extract and obtain face surface coordinate points of the patient's space sensed by the probe from image data of the patient's space in order to perform course matching. The medical image matching device may obtain a surface coordinate point indicating location information of the probe using a probe. The medical image matching device may acquire surface coordinate points of a part of the patient's face surface that is in contact with or adjacent to the probe. Also, the medical image matching device may generate the acquired surface coordinate points as the second coordinate points. For example, the second coordinate point may include a face surface coordinate point in the patient's space when the patient's forehead, nose tip, eye tip, and forehead are contacted.

In step S230, the medical image matching device may generate a course matching coordinate point by course matching the first coordinate point with the second coordinate point. Also, the apparatus for matching medical images may extract rotational and translational transformation matrices in case of course matching. Since the medical image matching device initially performs course matching, accuracy may be improved.

In operation S240, the medical image matching device may convert the face surface coordinate points in the patient's space into the course-matched surface coordinate points. The medical image registration device may generate a first registration coordinate point using the extracted rotation and translation matrix. Here, the facial surface coordinate points in the patient's space represent the positional information of the probe. The probe moves along the patient's skin surface in a contact or proximity state for a time of about 10 seconds and collects data on the patient's body surface. In one example, the probe may collect data for a period of 10 to 20 seconds.

The medical image matching device matches the face surface coordinate points in the patient's space with the coordinate points that are course-matched based on the extracted rotational and translational transformation matrices in the case of course matching. Also, the medical image matching device may set surface coordinate points, which are matching results, as first matching coordinate points in one spatial coordinate system.

In step S250, the medical image matching device may augment the first matching coordinate point. The step of augmenting the first registration coordinate point may be repeatedly performed a plurality of times. The first matching coordinate point may be augmented by 5%. Assuming that the process of augmenting the first registration coordinate point by 5% is one step, the medical image matching device may repeat one or more steps. The medical image matching device can increase matching accuracy by augmenting the number of coordinate points of the first matching coordinate points obtained within a limited time.

Also, the medical image matching device may match the augmented first matching coordinate point with the surface coordinate point of the medical image data. In this case, the medical image matching device may perform matching using an ICP algorithm. Also, the apparatus for matching medical images may derive rotational and translational transformation matrices in the case of performing the second matching.

In step S260, the medical image matching apparatus may analyze a registration error or registration accuracy through surface registration error analysis. The surface registration error is obtained by calculating and averaging the distance between one set of surface coordinate points and another set of surface coordinate points.

Also, the medical image matching apparatus may determine whether the analyzed surface registration error satisfies a preset end condition. In this case, the medical image matching device may repeatedly perform steps S250 and S260 when the surface registration error does not satisfy the end condition. That is, the apparatus for matching medical images may repeatedly perform the step of re-augmenting and matching the first matching coordinate points and the step of determining the surface matching error.

In step S270, the medical image matching device determines the surface coordinate points of the medical image data corresponding to the face surface coordinate points of the patient space and the final augmented patient space from the matching result when the surface matching error satisfies the preset end condition. It is possible to extract the face surface coordinate points of The extracted result means the second matching coordinate point.

5 is a diagram illustrating a method of building an artificial intelligence model according to an embodiment of the present invention.

First, in step S310, the artificial intelligence model may secure a data set based on the coordinate points of the lesion location of the phantom and the coordinate points of the facial surface in the patient's space. The phantom is a facial model, and may include a virtual lesion structure. In this case, the coordinate points of the lesion location of the phantom refer to a lesion candidate group arbitrarily assigned to the virtual lesion structure. For example, the coordinate points of the location of the lesion of the phantom may include coordinate points extracted from a CT image of the phantom. In addition, the artificial intelligence model may acquire facial surface coordinate points in the patient's space extracted from the probe through repetitive experiments. As will be described later, the medical image matching device may implement optimized registration by additionally including the coordinate points of the lesion location of the phantom in the artificial intelligence model.

In step S320, the input and output of the artificial intelligence model may be set. For example, the feature points of the input information may be set to the face surface coordinate points of the patient's space and the surface coordinate points of the medical image data corresponding to the coordinate points. In this case, the coordinate point is a 3D coordinate point and may be displayed in the form of (x, y, z).

In addition, feature points of the artificial intelligence model input information may additionally include lesion location coordinates, which are location information of lesions inside the phantom. Here, the lesion location coordinate point may include a location coordinate point of an arbitrarily designated lesion candidate group and a location coordinate point of a location corresponding to the lesion candidate group in the image obtained by capturing the phantom. At this time, the lesion location coordinates are used as feature points of artificial intelligence model input information.

Meanwhile, output information of the artificial intelligence model may be set to a rotation and translation matrix extracted through ICP matching.

In step S330, the artificial intelligence model divides the secured data set for learning and testing. Here, overfitting, which shows high predictive power only in the training data, can be prevented by dividing the training data set and the verification data set.

In addition, step S330 may be performed to secure sufficient training data to increase the learning performance of the artificial intelligence model. For example, the artificial intelligence model may augment training data by utilizing a jittering technique in which random noise is added and a scaling technique in which a data distribution of a data set is changed to be constant.

In step S340, the artificial intelligence model may estimate a rotation and translation matrix by learning the extracted feature points. For example, the artificial intelligence model may use a machine learning model that learns by extracting features of data to estimate rotational and translational transformation matrices. In this case, the machine learning model may be implemented as Long Short Term Memory (LSTM). LSTM is an artificial intelligence model that self-learns characteristic values over time and predicts them based on correlated time-series data. Here, the artificial intelligence model may be implemented as Bi-LSTM (Bidirectional LSTM) that can use not only forward but also backward results to improve accuracy.

In step S350, the learned artificial intelligence model may be tested. An artificial intelligence model trained with a training data set can be evaluated using a validation data set. For example, an artificial intelligence model may be tested by cross validation. Cross-validation is a verification method to find optimal parameters to prevent overfitting. When an AI model is tested with cross-validation, a portion of the training data set can be split into a validation data set. The artificial intelligence model partially alternates between the training data set and the verification data set to create multiple training data sets and verification data sets, and outputs the results for each data set created to be used to verify the artificial intelligence model. can In addition, the artificial intelligence model may estimate an optimized parameter using the divided verification data set, and modify the artificial intelligence model so that an error of the artificial intelligence model is reduced.

In addition, when the artificial intelligence model is finally formed, the artificial intelligence model may evaluate the artificial intelligence model using the test data set.

6 is a diagram illustrating input information for learning an artificial intelligence model according to an embodiment of the present invention.

As shown in FIG. 6, the face surface coordinate points in the patient space are set to (x, y, z) coordinates, and the surface coordinate points of the medical image data corresponding thereto are (x', y', z'). can be set. That is, the feature points of the input information may be set in six dimensions. In this case, when the length of the surface coordinate point of the medical image data is 150, the size of the input information may be set to 6 by 150.

In addition, coordinate points of the location of the lesion of the phantom may be additionally extracted from an image of the phantom. Here, the coordinate points of the lesion location of the phantom may include a location coordinate point of a randomly designated lesion candidate group and a location coordinate point of a location corresponding to the lesion candidate group in an image obtained by capturing the phantom. At this time, the lesion location coordinates are used as feature points of artificial intelligence model input information.

The extracted lesion location coordinates of the phantom consist of six dimensions and can be added to the input information. Similarly, when the length of the surface coordinate points of the medical image data is 150, the size of input information using the phantom may be set to 6 by 150. Accordingly, the final size of the input information may be set to 6 by 300.

7 is a diagram showing the Bi-LSTM architecture of an artificial intelligence model according to an embodiment of the present invention.

First, LSTM is one of RNN (Recurrent Neural Network) artificial intelligence models and is suitable for continuous time series data. The LSTM architecture is an artificial intelligence model that predicts correlated time series data, and is one of the recurrent neural network models that sequentially process data. There is a short-term storage layer (Short term, hidden state) and a layer responsible for long-term storage (Long term, cell state), so it has high accuracy in predicting time series data. Also, LSTM architectures are usually

In one embodiment, Table 1 is a table of learning options for an LSTM architecture. As shown in Table 1 and FIG. 7, the artificial intelligence model can configure a Stacked LSTM with a plurality of LSTMs. In addition, the artificial intelligence model can be implemented as a Bidirectional LSTM (Bidirectional LSTM) that can use not only forward but also backward results to improve accuracy. Layer 1 of Bi-LSTM is defined as Bi-LSTM#1, and layer 2 of Bi-LSTM is defined as Bi-LSTM#2.

Meanwhile, a data set based on the coordinate points of the lesion location of the phantom and the coordinate points of the facial surface of the patient space may be input to the artificial intelligence model. For example, a face surface coordinate point in a patient space may be set to (x, y, z) coordinates, and a surface coordinate point of medical image data corresponding thereto may be set to (x', y', z'). That is, input information can be set in six dimensions.

In addition, bidirectional learning can be implemented in Bi-LSTM#1 and Bi-LSTM#2 of the artificial intelligence model. Result values output from each Bi-LSTM layer may be combined in a concatenate layer.

In addition, result values combined in the connection layer may be input to a fully connected layer. A fully connected layer is a layer that determines classification through a flattened matrix in the form of a one-dimensional array. That is, among the result values input to the fully connected layer, some result values may be extracted. Here, the fully connected layer may be implemented in a drop out form to prevent overfitting. At this time, the dropout ratio may proceed to 0.2, and other specific result values may be output according to the dropout ratio. In addition, if the artificial intelligence model is complex, the dropout rate can be increased, and if it is simple, the dropout rate can be adjusted by decreasing it.

In addition, rotation and translation matrix values may be output through a regression layer. The regression layer predicts the correlation between the dependent variable and the independent variable. In addition, the rotation matrix of the rotation and translation matrix values may be output in a 3 by 3 (R) dimension and the translation matrix in a 3 by 1 (T) dimension. Accordingly, the rotation and translation matrix values output through the artificial intelligence model may include 12 dimensions.

Since the artificial intelligence model is optimized by including the phantom lesion location information, the output rotation and translation matrix can be learned in a way that can reduce the amplification of the deep lesion error.

Table of training options for LSTM architecture No. Type of parameters Range of parameters Range of parameters One Number of hidden units [200,100] [100,50] 2 Maximum epochs 100 100 3 Mini-batch size 64 32 4 Weight initializer function Glorot Glorot 5 Solver Adam Adam 6 Drop out rate 0.2 0.2 7 Initial learning rate 0.01 0.01 8 Gradient threshold 2 2 9 Gradient threshold method Global-l2norm Global-l2norm 10 L2Regularization 1e^-5 1e^-6 11 Sequence padding shortest longest 12 Sequence output mode sequence sequence

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form. The scope of the present invention is set forth in the claims that follow. It is indicated by, and all changes or modified forms derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention.

110. Input unit
120. Coordinate point generator
130. Augmentation
140. Artificial intelligence model storage unit
150. Matching part
160. Output unit

Claims (12)

A medical image matching device for precisely matching image data of a patient's space and medical image data in order to improve endoscopic augmented reality precision and reduce errors in deep lesions,
an input unit receiving image data of the patient's space and the medical image data;
an artificial intelligence model storage unit storing an artificial intelligence model for estimating a rotational and translational transformation matrix value for precision matching based on the image data of the patient's space and the medical image data;
a matching unit that precisely matches the image data of the patient's space and the medical image data; and
and an output unit outputting a matching result of the image data of the patient space and the medical image data.
The matching unit may include a first matching unit generating a first matching coordinate point by roughly matching the image data of the patient space and the medical image data;
a second matching unit generating a second matching coordinate point by matching the first matching coordinate point with the surface coordinate point of the medical image data; and
and a third matching unit for precision matching based on the rotation and translation matrix values estimated by taking the second matching coordinate point as an input of the artificial intelligence model.
According to claim 1,
The input information feature points of the artificial intelligence model include face surface coordinate points in the patient space and surface coordinate points of the medical image data corresponding to the facial surface coordinate points in the patient space, and are set in six dimensions. A medical image matching device.
According to claim 2,
The artificial intelligence model further secures a data set based on the coordinates of the lesion location of the phantom to set input feature points of the artificial intelligence model,
The input feature points of the artificial intelligence model based on the lesion location coordinate points are the location coordinate points of the lesion candidate group arbitrarily designated in the virtual lesion structure and the location coordinates corresponding to the location information of the lesion candidate group in the image data obtained by photographing the lesion structure. A medical image matching device that includes points and is set in six dimensions.
According to claim 1,
The artificial intelligence model includes a Bidirectional LSTM (Bi-LSTM) architecture,
A medical image registration device that outputs a rotation and translation matrix as a 3X3-dimensional rotation matrix and a 3X1-dimensional translation matrix using a regression layer.
According to claim 1,
The second matching unit, the artificial intelligence model, and the third matching unit are matched using an iterative closest point (ICP) algorithm.
In the medical image matching method of precisely matching image data of patient space and medical image data to improve endoscopic augmented reality accuracy and reduce deep lesion errors,
a) acquiring image data of the patient's space and the medical image data;
b) roughly extracting and matching coordinate points from the image data of the patient space and the medical image data, and extracting a rough rotation and translation transformation matrix.
c) first matching the coarsely matched coordinate points and facial surface coordinate points extracted from the patient space image data using the extracted coarse rotation and translation matrix, and generating a first matching coordinate point; ;
d) augmenting the first registration data and generating a second registration coordinate point by matching the augmented first registration coordinate point with a surface coordinate point extracted from the medical image data; and
e) precision matching based on the rotation and translation matrix values estimated by taking the second registration coordinate point as an input of the artificial intelligence model;
According to claim 6,
The input information feature points of the artificial intelligence model include face surface coordinate points in the patient space and surface coordinate points of the medical image data corresponding to the facial surface coordinate points in the patient space, and are set in six dimensions. A method for matching medical images.
According to claim 2,
The artificial intelligence model further secures a data set based on the coordinates of the lesion location of the phantom to set input feature points of the artificial intelligence model,
The input feature points of the artificial intelligence model based on the lesion location coordinate points are the location coordinate points of the lesion candidate group arbitrarily designated in the virtual lesion structure and the location coordinates corresponding to the location information of the lesion candidate group in the image data obtained by photographing the lesion structure. A medical image matching method that includes points and is set in six dimensions.
According to claim 6,
The artificial intelligence model includes a Bidirectional LSTM (Bi-LSTM) architecture,
A medical image matching method comprising outputting a rotational and translational transformation matrix as a 3X3-dimensional rotational transformation matrix and a 3X1-dimensional translational transformation matrix using a regression layer.
According to claim 6,
Steps d) and steps e) perform matching using an iterative closest point (ICP) algorithm.
According to claim 6,
Step d) augments the first matching coordinate point by 5% for each augmentation;
The method of matching medical images, wherein the augmentation is performed a plurality of times.
According to claim 11,
The step d) includes analyzing a surface matching error after each augmentation is completed,
and stopping augmentation when the surface registration error analysis value satisfies a preset end condition.

Images