KR102580750B1 - 3d image registration method based on markerless, method for tracking 3d object and apparatus implementing the same method - Google Patents

Abstract

The present invention relates to a markerless-based 3D image registration method and a 3D object tracking method and device using the same. It is a method of operating a computing device operated by at least one processor, and is a method of operating a computing device operated by at least one processor. Inputting a 2D soft tissue image into a learned feature point extraction model to extract one or more first feature points based on anatomical location, and reconstructing the first feature point into 3D coordinates based on the medical image; detecting the user with a depth recognition camera Inputting the captured image into a learned feature point extraction model to extract one or more second feature points based on anatomical location, applying depth information of the captured image to reconstruct the second feature point into three-dimensional coordinates, and It includes the step of matching the medical image and the captured image by matching the 3D coordinates of the first feature point and the 3D coordinates of the second feature point.

Description

Markerless-based 3D image registration method and 3D object tracking method and device using the same {3D IMAGE REGISTRATION METHOD BASED ON MARKERLESS, METHOD FOR TRACKING 3D OBJECT AND APPARATUS IMPLEMENTING THE SAME METHOD}

A markerless-based 3D image registration and tracking technology is provided.

Recently, minimally invasive surgery (MIS), which performs surgery with minimal incisions, has been developed and applied in medical surgical fields such as dentistry, otolaryngology, orthopedics/neurosurgery.

Minimally invasive surgery has fewer damaged areas, lower incidence of pain and complications, and faster recovery and faster return to daily life, but the surgical technique is more complicated than general open surgery. In particular, since it is difficult to accurately position the affected area or structure due to limited visibility in minimally invasive surgery, an image-guided surgical navigation system that combines a 3D position tracking device is used.

In particular, the augmented reality-based image-guided surgical navigation system using 3D medical images such as MRI/CT/ultrasound has a matching step between the medical image coordinate system captured before the patient's surgery and a physical coordinate system based on images taken during the surgery, and even after the matching step. Technology is essential to track the position of a moving tool in a physical coordinate system by attaching a position tracking device to the tool.

Accordingly, the existing registration method contacts a specific device (maker) mounted on the patient's body or an actual part of the patient's body by scratching one or more points, and matches the medical image coordinate system and the physical coordinate system based on the contact point.

In this case, the surgical area is obscured due to the volume of the specific instrument used separately, the accuracy of registration is determined by the technician's skill level, which increases the surgical preparation time due to errors, and causes friction between patients who come into contact with the patient's registration area. There is a possibility that cross-infection may occur.

Therefore, it is possible to automatically and non-contactly match the image coordinate system with the physical coordinate system without any additional tools, and technology is required to track the patient with high precision and high speed in the matched coordinates.

One embodiment of the present invention extracts the coordinates of anatomical feature points from medical images and captured images through a machine learning-based model without separate equipment such as markers, and matches the medical images and captured images based on the coordinates of the extracted feature points. And, based on continuously captured images, it provides a method of tracking objects in the image through corresponding feature point coordinates.

One embodiment of the present invention provides a 3D object tracking method and device that performs accurate object tracking at high speed by using a machine learning model and an optical tracking algorithm together.

In addition to the above tasks, embodiments according to the present invention can be used to achieve other tasks not specifically mentioned.

According to an embodiment of the present invention, as a method of operating a computing device operated by at least one processor, a two-dimensional soft tissue image generated based on a user's medical image is input into a learned feature point extraction model to extract data based on anatomical location. Extracting one or more first feature points and reconstructing the first feature points into three-dimensional coordinates based on the medical image, inputting the captured image of the user with a depth recognition camera into a feature point extraction model, one based on the anatomical location Extracting the above second feature point, applying the depth information of the captured image to reconstruct the second feature point into 3D coordinates, and matching the 3D coordinates of the first feature point with the 3D coordinates of the second feature point to produce a medical image. and matching the captured images.

The feature point extraction model can convert the input image into an image based on gradient information and then output two-dimensional coordinates of pre-labeled feature points from the image based on gradient information.

In the case of an image including a face, the feature point extraction model can select and label one or more points among the outer end point of the eye, the inner end point of the eye, the corner of the mouth, and the start point of the nose, where movement according to the face position is minimized, as a feature point.

The step of reconstructing the first feature point into 3D coordinates involves rendering into a 3D soft tissue model based on the medical image and generating a 2D soft tissue image projected from the 3D soft tissue model toward the coronal plane. This can be input into the feature point extraction model.

The step of reconstructing the first feature point into three-dimensional coordinates involves acquiring the two-dimensional coordinates of the first feature point for the two-dimensional soft tissue image, and using the two-dimensional coordinates of the first feature point as a reference point to create a three-dimensional model that has a contact point with the three-dimensional soft tissue model. Coordinates can be extracted.

The step of matching captured images involves performing point-to-point matching between feature points whose labeling matches the 3D coordinates of the first feature point and the 3D coordinates of the second feature point, and converting coordinates between the matched feature points. The value can be calculated.

In the step of matching the captured image, a coordinate transformation value may be applied to the medical image to convert the coordinate system of the medical image into a coordinate system for the captured image.

According to an embodiment of the present invention, there is provided a method of operating a computing device operated by at least one processor, comprising: collecting captured images continuously captured of a user through a depth recognition camera; and a feature point extraction model learned from the captured images. and selectively using an optical tracking algorithm to extract one or more labeled feature point coordinates for each captured image, and reconstructing the labeled feature point coordinates into 3D coordinates based on the depth information for each captured image, respectively, in the collection order. Accordingly, it includes the step of tracking an object having a feature point in a captured image by matching the 3D coordinates of the labeled feature point between adjacent captured images.

It further includes sequentially grouping the captured images based on a preset unit, and the step of extracting feature point coordinates includes extracting feature point coordinates for the first captured image among the captured images using a learned feature point extraction model, From the second captured image, feature point coordinates can be extracted using an optical tracking algorithm based on feature point coordinates extracted from the previous captured image.

The feature point extraction model can convert the input image into an image based on gradient information and then output two-dimensional coordinates of feature points pre-labeled based on anatomical positions in the image based on gradient information.

The step of tracking an object with a feature point involves performing point-to-point matching of the 3D coordinates of feature points with the same labeling between adjacent captured images according to the collection order, and 3D coordinates between the matched 3D coordinates. Dimensional change can be calculated.

Collecting the user's medical image with feature points extracted based on the anatomical position, matching it to the coordinate system of the captured image, and applying the calculated 3-dimensional change amount to the medical image to create a medical image corresponding to the captured image. A generating step may be further included.

According to one embodiment of the present invention, it includes a communication device, a memory, and at least one processor that executes instructions of a program loaded in the memory, and the program captures a user's medical image and a depth recognition camera. When the user's captured image is input into the learned feature point extraction model to obtain the coordinates of one or more first feature points and the coordinates of the second feature point, each labeled based on anatomical location, the coordinates of the first feature point and the coordinates of the second feature point are obtained. If the medical image and the captured image are matched through matching between the captured images, and the coordinates of one or more feature points are extracted for each captured image by selectively using a feature point extraction model and an optical tracking algorithm from the collected continuous captured images, the consecutive captured images Includes instructions that match feature point coordinates between the camera, calculate the amount of change in the matched feature point coordinates, and track the object in the captured image.

The program renders a 3D soft tissue model based on medical images, creates a 2D soft tissue image projected from the 3D soft tissue model toward the coronal plane, and uses a feature point extraction model to create a 2D soft tissue image. Instructions may be included to obtain the 2-dimensional coordinates of the first feature point for the soft tissue image and extract 3-dimensional coordinates having a contact point with the 3-dimensional soft tissue model using the 2-dimensional coordinates of the first feature point as a reference point. there is.

The program performs point-to-point matching between feature points whose labeling matches the 3-dimensional coordinates of the first feature point and the 3-dimensional coordinates of the second feature point reconstructed by applying the depth information of the captured image, and matching Instructions described to calculate coordinate transformation values between feature points may be included.

The program groups sequentially captured images sequentially based on preset units, extracts feature point coordinates using a learned feature point extraction model for the first captured image in the group, and extracts feature point coordinates from the immediately preceding image for the second captured image. It may include instructions to extract feature point coordinates using an optical tracking algorithm based on the feature point coordinates.

According to one embodiment of the present invention, registration is performed between 3D images in an unconstrained manner using anatomical feature points without using markers and tracking objects, thereby providing fast and accurate data with a minimized amount of calculation without the need for additional equipment such as markers. Matching results and tracking data can be obtained.

According to one embodiment of the present invention, since the registration of medical images and depth images is performed automatically in real time, it is not affected by the technical skill of the technician for 3D image registration, and thus errors due to differences in skill are minimized and accuracy is maintained at a constant level. High matching results can be obtained.

According to one embodiment of the present invention, the object can be tracked at high speed while preventing the occurrence of cumulative errors by simultaneously tracking the object using a machine learning model and an optical tracking algorithm based on the anatomical feature points of the object being tracked.

According to one embodiment of the present invention, surgical accuracy can be improved in image-guided surgery by providing quick and accurate registration between images and object tracking in images based on anatomical features of the patient.

Figure 1 is a configuration diagram showing a computing device for 3D image registration and object tracking according to an embodiment of the present invention.
Figure 2 is a flowchart showing a method of operating a computing device according to an embodiment of the present invention.
Figure 3 is a flowchart showing a method of operating a computing device according to another embodiment of the present invention.
Figure 4 is an example diagram for explaining the operation of a feature point extraction model according to an embodiment of the present invention.
Figure 5 is an exemplary diagram illustrating a process of reconstructing the coordinates of feature points extracted from a medical image in three dimensions according to an embodiment of the present invention.
Figure 6 is an example diagram to explain the process of reconstructing the coordinates of feature points extracted from a captured image in three dimensions according to an embodiment of the present invention.
Figure 7 is an example diagram to explain the process of extracting feature point coordinates from continuously captured images according to an embodiment of the present invention.
Figure 8 is an example diagram for explaining a tracking process through matching feature point coordinates according to an embodiment of the present invention.
Figure 9 is a hardware configuration diagram of a computing device in one embodiment of the present invention.

With reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. The present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts not related to the description are omitted, and the same reference numerals are used for identical or similar components throughout the specification. Additionally, in the case of well-known and well-known technologies, detailed descriptions thereof are omitted.

Throughout the specification, when a part is said to “include” a certain element, this means that it may further include other elements rather than excluding other elements, unless specifically stated to the contrary.

In addition, terms such as “…unit”, “…unit”, and “…module” used in the specification refer to a unit that processes at least one function or operation, which may be implemented through hardware or software or a combination of hardware and software. You can.

The device described in the present invention is composed of hardware including at least one processor, memory device, communication device, etc., and a program to be executed in conjunction with the hardware is stored in a designated location. The hardware has a configuration and performance capable of executing the method of the present invention. The program includes instructions that implement the operating method of the present invention described with reference to the drawings, and executes the present invention by combining it with hardware such as a processor and memory device.

In this specification, “transmission or provision” may include not only direct transmission or provision, but also indirect transmission or provision through another device or using a circuitous route.

In this specification, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component without departing from the scope of the present disclosure.

In the embodiments described herein with reference to the drawings, any embodiment may be implemented independently, several embodiments may be merged or divided, and specific operations may not be performed in each embodiment.

1 is a configuration diagram showing a computer device for 3D image registration and object tracking according to an embodiment of the present invention.

As shown in FIG. 1, the computer device 100 includes an image registration module 110 and an object tracking module 120, and may further include a learning module 130.

For explanation purposes, they will be referred to as the image registration module 110, the object tracking module 120, and the learning module 130, but these are computing devices operated by at least one processor. Here, the image registration module 110, the object tracking module 120, and the learning module 130 may be implemented in one computing device, or may be distributed and implemented in separate computing devices. When distributed and implemented in separate computing devices, the image registration module 110, the object tracking module 120, and the learning module 130 may communicate with each other through a communication interface. The computing device may be any device capable of executing a software program written to carry out the present invention, and may be, for example, a server, a laptop computer, etc.

For convenience of explanation, the modules are classified into the image matching module 110 and the object tracking module 120. However, components that perform the same function in each module are not provided separately and one component may be shared with each other.

The image registration module 110 collects a medical image (A) and a captured image (B) captured through a depth recognition camera.

The image registration module 110 may collect medical images linked to the user's ID by accessing a device that generates medical images or a database where medical images are stored.

Here, medical images are three-dimensional images captured through medical devices, such as magnetic resonance imaging MRI, computed tomography (CT), positron emission tomography (PET), and ultrasound imaging. indicates.

The image matching module 110 can collect captured images of a user in real time by linking with a depth recognition camera or a storage device of a depth recognition camera.

Here, the depth recognition camera (not shown) is a triangulation-type 3D laser scanner, an endoscope device, a depth camera using a structural light pattern, and a time-of-flight (TOF) camera using the difference in reflection time of infrared (IR: Infra-Red). -flight) type depth camera, C-arm device, optical coherence tomography, etc.

When a depth recognition camera captures an image, it simultaneously creates a color image and a depth image, so the captured image includes both a color image and a depth image.

The image registration module 110 extracts preset labeled feature points for each collected image.

Here, the labeled feature point is a preset specific point located on soft tissue, but refers to a point where movement is minimized due to anatomical structure or analysis.

For example, the points of minimal movement in soft tissue locations of the face include the exocanthion (left, right), the inner endpoint of the eye (endocanthion (left, right)), the corner of the mouth (cheilion (left, right)), and It can be set by specifying and labeling 7 points, such as the nose starting point (Pronasale).

Accordingly, the image registration module 110 can extract labeled points from the collected images as feature points.

Meanwhile, the image registration module 110 can extract feature points from medical images or captured images using a learned feature point extraction model, and when extracting feature points from continuously captured images, the learned feature point extraction model and optical tracking algorithm are used. Can be used simultaneously.

The image registration module 110 reconstructs feature point coordinates for the medical image by applying each feature point extracted from the medical image or captured image, and reconstructs feature point coordinates for the captured image.

In other words, the image registration module 110 converts the feature points into 3D location information because they are extracted as 2D locations.

In detail, the image registration module 110 sets the extracted feature point as a reference point in the medical image, extracts the coordinates of the medical image having a contact point with the reference point, and uses the depth information included in the captured image to locate the corresponding two-dimensional location. Depth information can be applied and converted to 3D coordinates.

The image registration module 110 performs registration by matching the feature point coordinates of the reconstructed medical image and the feature point coordinates of the reconstructed captured image.

Here, the image registration module 110 may perform registration to convert the coordinate system of the medical image into the coordinate system of the captured image through matching between feature point coordinates.

The object tracking module 120 collects continuous captured images (B) in conjunction with a depth recognition camera or a storage device of a depth recognition camera.

The object tracking module 120 may collect real-time captured images while grouping captured images based on preset units.

The object tracking module 120 selectively applies a feature point extraction model and an optical tracking algorithm learned based on the order of grouped captured images to extract two-dimensional coordinates of a preset labeled feature point for each captured image.

Then, the object tracking module 120 uses the depth information included in each captured image to apply the depth information to the corresponding 2-dimensional location and convert it into 3-dimensional coordinates.

The object tracking module 120 calculates a 3D position change amount by matching the 3D coordinates of feature points between images captured in the order of time for consecutive captured images. At this time, the object tracking module 120 may apply the calculated 3D position change amount to the registered medical image and reflect the result of tracking the object in the real-time captured image in the medical image.

And the learning module 130 learns a feature extraction model that extracts preset anatomical feature points (labeled feature points) from the input image.

The feature point extraction model is a machine learning model that converts the input image into an image based on gradient information and then extracts the coordinates of the feature points labeled based on the gradient information.

The feature point extraction model can be implemented with machine learning algorithms such as Support Vector Machine (SVM), Random Forest (RF), and Convolution Neural Network (CNN).

In this way, the learning module 130 can train a feature point extraction model inside the computing device 100, but the learning module 130 can provide the computing device 100 with a feature point extraction model that has been trained in a separate device. .

Meanwhile, the feature point extraction model extracts the positions of pre-labeled feature points from a two-dimensional image. The same feature point extraction model can be used in the image registration module 110 and the object tracking module 120, but depending on the type of input image, It can be divided into a feature point extraction model for medical images and a feature point extraction model for captured images.

Hereinafter, using FIGS. 2 and 3, a detailed description will be given of how a computing device performs registration between 3D images and a method of tracking an object in a 3D image.

Figure 2 is a flowchart showing a method of operating a computing device according to an embodiment of the present invention.

As shown in FIG. 2, the computing device 100 collects the user's medical image (S110). Then, the computing device 100 generates a two-dimensional soft tissue image based on the medical image (S120).

The computing device 100 may generate a 3D soft tissue model by performing 3D rendering on a 3D medical image and generate a 2D soft tissue image from the generated 3D soft tissue model.

Here, the soft tissue model and soft tissue image refer to a model and image of soft tissue that has characteristics of low rigidity, such as a face model or face image.

In detail, the computing device 100 generates a 3D soft tissue model of a medical image using a surface rendering algorithm such as a marching cube. Then, the computing device 100 projects the 3D soft tissue model onto the XZ plane (coronal plane) to generate a 2D soft tissue image.

Here, the 3D soft tissue model and the 2D soft tissue image generated by the computing device 100 are both based on medical images and are identical to the coordinate system of the medical image (coordinate system of the medical imaging device).

Since the coordinate system of the medical imaging device is aligned with the global horizontal, the computing device 100 projects to the XZ plane, which is the coronal plane direction, so that the front part of the face is represented when the soft tissue is a facial area.

At this time, although FIG. 2 shows collecting medical images and generating a two-dimensional soft tissue image, the computing device 100 collects only the user's medical image itself, or collects a three-dimensional medical image for the user, a three-dimensional soft tissue model, and 2 One or more medical images may be collected from the dimensional soft tissue images.

For example, the computing device 100 may collect a 3D soft tissue model or a 2D soft tissue image generated based on a 3D medical image from another device.

Next, the computing device 100 extracts two-dimensional feature points using the learned feature point extraction model (S130).

When the computing device 100 receives a two-dimensional soft tissue image for a medical image, it inputs it into a learned feature point extraction model for extracting feature points. And the computing device 100 extracts the two-dimensional coordinates (P ⁱ _n ) of the first feature point from the learned feature point extraction model. (P ⁱ _n is the 2-dimensional coordinate of the feature point in the 2-dimensional soft tissue image, n is the number of feature points, and i represents the medical image)

In detail, the computing device 100 converts a two-dimensional soft tissue image into a gradient information-based image through a learned feature point extraction model and extracts a labeled first feature point from the tilt information-based image.

Then, the computing device 100 converts the two-dimensional soft tissue image into a gradient information-based image through a feature point extraction model learned for the color image of the captured image, and extracts the labeled first feature point from the tilt information-based image.

Here, the output value of the learned feature point extraction model is the coordinates of each labeled feature point.

Next, the computing device 100 reconstructs the two-dimensional coordinates of the feature points extracted based on ray projection into three-dimensional coordinates (S140).

The computing device 100 projects the coordinates of the first feature point in the two-dimensional soft tissue image onto the two-dimensional soft tissue model and reconstructs them into three-dimensional coordinates.

At this time, the computing device 100 may project a ray from the coordinates of each feature point using a ray projection algorithm and then extract the 3D coordinates of the 3D soft tissue model forming the contact point.

In other words, the computing device 100 projects a light ray from a 2-dimensional soft tissue image to a 3-dimensional soft tissue model using the 2-dimensional coordinates (P ⁱ _n ) of the first feature point as a reference point, and creates a 3-dimensional ray that forms the first contact point of the projected ray. Extract the 3D coordinates f(P ⁱ _n ) of the soft tissue model. (f represents the ray casting algorithm)

Through this, the computing device 100 can reconstruct from the two-dimensional coordinates (P ⁱ _n ) of the first feature point to the three-dimensional coordinates f (P ⁱ _n ) of the first feature point.

Meanwhile, steps S110 to S140 described above are performed in real time or at a previous time, and after performing each step and storing the results, the computing device 100 collects the stored 3D coordinates before performing registration between images. You can.

In other words, steps S110 to S140 are not necessarily performed in real time, and the execution timing may vary depending on the applied environment.

Next, the computing device 100 collects the user's captured images (S150).

Here, the captured image is captured by a depth imaging camera and includes a two-dimensional color image and a depth image.

Then, the computing device 100 extracts two-dimensional feature points using the learned feature point extraction model (S160).

The computing device 100 converts the color image of the captured image into a gradient information-based image through the learned feature point extraction model and extracts labeled feature points from the tilt information-based image.

In other words, the computing device 100 inputs a two-dimensional color image into the learned feature point extraction model and extracts the two-dimensional coordinates (P ^c _n ) of the second feature point from the two-dimensional color image. (P ^c _n is the 2-dimensional coordinate of the feature point in the 2-dimensional color image, n is the number of feature points, and c represents the captured image)

Next, the computing device 100 reconstructs the two-dimensional coordinates of the feature points extracted based on depth information into three-dimensional coordinates (S170).

The computing device 100 reconstructs the 3D coordinates p (P _n ^c ) using the depth image information of the 2D coordinates (P _n ^c ) of the second feature point. (p indicates pinhole camera model)

This can be expressed as Equation 1 below:

[Equation 1]

Here, p (P ^c _n ) represents the three-dimensional coordinates of the second feature point, f _x and fy are the focal length of the depth image camera, and c _x and c _y are the camera principal points. It means internal parameter.

Next, the computing device 100 performs registration based on the 3D coordinates of the reconstructed feature points (S180).

The computing device 100 calculates the three-dimensional coordinates f (P ⁱ _n ) of the second feature point for the medical image and the three-dimensional coordinate p (P ^c _n ) of the second feature point for the captured image in a point-to-point manner. ) Perform matching to perform 3D registration.

For example, the computing device 100 divides the 3D coordinates of the 7 labeled first feature points extracted from the medical image and the 3D coordinates of the 7 labeled second feature points extracted from the captured image into point-to-point based on the same labeling. Perform point matching.

The computing device 100 performs point-to-point matching between the 3D coordinates of the first feature point and the 3D coordinates of the second feature point.

Then, the computing device 100 calculates coordinate conversion information from the medical image to the captured image (S190).

The computing device 100 calculates a transformation value for converting the 3D coordinate system of the medical image into the 3D coordinate system of the captured image through matching between feature point coordinates. Through this, the computing device 100 can apply the conversion value to the entire medical image to match the medical image and the captured image.

Through this, the computing device 100 performs point-to-point matching of the 3D coordinates of the seven feature points and calculates a conversion value (T ^c _i ) from the coordinate system of the medical image to the coordinate system of the captured image.

If this is expressed in a mathematical equation, it is as follows in equation 2.

[Equation 2]

p(P ^c _n )= T ^c _{i *} f(P ⁱ _n )

The computing device 100 can convert the entire medical image into the coordinate system of the captured image using a conversion value (T ^c _i ) and stores the conversion value (T ^c _i ).

Figure 3 is a flowchart showing a method of operating a computing device according to another embodiment of the present invention.

As shown in FIG. 3, the computing device 100 collects continuously captured images and groups them into certain units based on order (S210).

The computing device 100 may collect captured images in real time and group the captured images into certain units.

For example, when set to 8 frames, the computing device 100 groups continuously collected images in units of 8 frames.

At this time, in order to selectively apply the feature point extraction model and optical tracking algorithm learned based on the order of the grouped captured images, the computing device performs grouping based on the order of consecutive captured images.

Next, the computing device 100 extracts the two-dimensional coordinates of the feature point using a feature point extraction model for the first captured image based on the position within the group (S220).

The computing device 100 extracts the two-dimensional coordinates (P ^c M _n ) of the feature point from the first image using a feature point extraction model. (P ^c M _n represents the two-dimensional coordinates of the feature point of the Mth acquired color image, M is a natural number)

It is desirable for the computing device 100 to extract all preset labeled feature points from the captured image, but because the shape in the captured image changes due to the user's movement, it extracts at least three labeled feature points located in the captured image. Here, the three labeled feature points are the minimum number of feature points for applying 3D transformation, and are not necessarily limited thereto.

Here, the step of extracting feature points by inputting the captured image into the feature point extraction model is the same as step S160, so redundant explanation will be omitted.

Next, the computing device 100 extracts the two-dimensional coordinates of each feature point through an optical tracking algorithm based on the two-dimensional coordinates of the feature point of the previous shot from the second shot image to the last shot image in the group (S230).

The computing device 100 determines two-dimensional feature points (P ^c M + 1 _n = O (P ^c M _n )) is extracted. (P ^c M _n is the two-dimensional coordinate of the feature point of the M-th acquired color image, P ^c M+1 _n is the two-dimensional coordinate of the feature point of the M+1-th acquired color image, O represents the optical tracking algorithm)

The computing device 100 reconstructs the two-dimensional coordinates of the feature points extracted based on depth information for each captured image into three-dimensional coordinates (S240).

And the computing device 100 reconstructs the captured image from the two-dimensional coordinates (P ^c M _n ) of the feature point to the three-dimensional coordinate p (P ^c M _n ) based on the depth image information and the pinhole camera model in the same manner as in step S170. do.

Next, the computing device 100 tracks the location of the object in the captured image by matching the 3D coordinates of the feature points reconstructed in order (S250).

The computer device 100 matches and tracks the three-dimensional coordinates of identically labeled feature points between consecutive captured images.

In detail, the 7 feature points are matched point-to-point based on labeling by matching the 3D coordinates between feature points labeled as the outer endpoint of the eye.

Then, the computing device 100 calculates the amount of change in the 3D coordinates of the feature points between captured images (S260).

If this is expressed in a mathematical equation, it is as follows in equation 3.

[Equation 3]

p(P ^c M+1 _n )= T _M ^M+1 _* p(P ^c M _n )

The computing device 100 may apply a change amount (T _M ^M+1 ) to the M-th captured image to convert it to correspond to the M+1-th captured image, and store the corresponding change amount (T _M ^M+1 ).

The computing device 100 performs point-to-point matching and variation (T _M ^{M + 1} ) calculations on the 3-dimensional coordinates of 7 feature points for two consecutive captured images, and converts the matching transformation value (T ^c _i ) into 3-dimensional By applying it to captured images, it can provide 3D medical images corresponding to each captured image.

If this is expressed in a mathematical equation, it is as follows in equation 4.

[Equation 4]

f(P ⁱ M+1 _n )= (T ^c _i ) ^-1 _* T _M ^M+1 _* p(P ^c M _n )

In other words, it is possible to provide a medical image to which changes in feature point coordinates between captured images are applied by applying the change amount and registration change amount of the tracking point in the camera coordinate system.

Equation 4 estimates the feature point in the m+1th medical image from the feature point of the mth captured image, and vice versa is also possible.

Figure 4 is an example diagram for explaining the operation of a feature point extraction model according to an embodiment of the present invention.

As shown in FIG. 4, the computing device 100 converts the input image into a gradient information image (HOG feature, Histogram of Oriented Gradients) and uses a feature point extraction model (SVM and Random forest) is used.

For example, the computing device 100 removes non-essential information (e.g., background of a certain color) from the input two-dimensional image, converts it into a gradient image in which outlines are emphasized, and minimizes the influence of lighting changes. Normalization can be performed to generate the final gradient information image.

Accordingly, the computing device 100 may input the tilt information image into a feature point extraction model and extract pre-labeled feature points from the corresponding tilt information image.

Figure 5 is an exemplary diagram showing a process of reconstructing the coordinates of feature points extracted from a medical image in three dimensions according to an embodiment of the present invention.

Figure 6 (a) shows the two-dimensional coordinates (A-4) of the feature point extracted from the medical image (A-1), and in (b), the two-dimensional coordinates of the extracted feature point are converted into three-dimensional coordinates.

As shown in (a) of FIG. 6, in order to input a feature point extraction model from a medical image (A-1) captured by a medical device, the computing device 100 renders the medical image (A-1) in three dimensions. After creating a 3D soft tissue model (A-2), a 2D soft tissue image (A-3) can be created by projecting it to the XZ plane.

However, as described above, the computing device 100 can collect one or more medical images among a medical image (A-1), a three-dimensional soft tissue model (A-2), and a two-dimensional soft tissue image (A-3). , Depending on the type of medical image collected, it can be input directly into the feature extraction model, either through the image creation process or without.

And as shown in (b) of FIG. 5, the computing device 100 projects a ray from the 2-dimensional coordinates (A-4) of the feature point to the 3-dimensional soft tissue model (A-2) and reconstructs it into 3-dimensional coordinates for the first contact point. can do. Through this process, the two-dimensional coordinates of the feature points extracted from the medical image are converted into three-dimensional coordinates.

Figure 6 is an example diagram to explain the process of reconstructing the coordinates of feature points extracted from a captured image in three dimensions according to an embodiment of the present invention.

As shown in FIG. 6, the captured image B has a two-dimensional color image and a two-dimensional depth image, and the computing device 100 extracts two-dimensional coordinates of labeled feature points from the two-dimensional color image.

Since the 2D color image and the 2D depth image have the same coordinate system, the computing device 100 can obtain the corresponding depth information by extracting the 2D coordinates of the feature point from the 2D depth image.

Accordingly, the computing device 100 can reconstruct the 3D coordinates of the 7 feature points by applying the depth information of the point based on the 2D depth image of the captured image to the 2D coordinates of the feature points.

Hereinafter, the process of tracking an object in continuously captured images will be described in detail using FIGS. 7 and 8.

Figure 7 is an example diagram to explain the process of extracting feature point coordinates from continuously captured images according to an embodiment of the present invention.

As shown in FIG. 7, the computing device 100 extracts feature points from continuously captured images collected over time by using a machine learning model and a scientific tracking algorithm together.

In Figure 7, the frames #1 to #M (for example, 8 frames) are assumed to be one group, and the process of extracting feature points for the frames collected within the group is explained based on time. (M is a natural number)

For convenience of explanation, if one frame is received at each time point, when the computing device 100 receives frame #1 at time t ₀ , it extracts a landmark through a learned feature point extraction model, which is a machine learning model. Extract. Here, as previously explained, the feature point refers to a point where movement is minimized based on the anatomical structure.

The computing device 100 may extract and simultaneously store the feature point coordinates for frame #1.

Since the number of grouped frames is assumed to be 8, when the computing device 100 receives consecutive frames from time t ₀ to time t ₇ , they temporarily store them in a buffer (save buffer).

Accordingly, the computing device 100 extracts the feature point (#2 Landmark) from the #2 frame using an optical tracking algorithm (Optical flow) based on the feature point (#1 Landmark) extracted from the #1 frame stored at t ₈ . Extract and save.

Then, based on the feature points extracted from the previous frame from time t ₉ to time t _14, the feature points for the frame are extracted and stored using an optical tracking algorithm.

Meanwhile, one group is described in FIG. 7, but when applied to real-time captured images, feature points can be extracted for a plurality of groups as described above by grouping successive captured images in arbitrary units.

At this time, for real-time purposes, captured images belonging to other groups can be collected and feature points extracted at the same time as the feature point extraction process for one group.

For example, after receiving a frame for one group from time t ₀ to t ₇ , the feature point coordinates for the second frame can be extracted at time t ₈ and the first frame from another group can be received at the same time.

This configuration can be displayed as shown in Table 1 below.

Current Image M M+1 M+2 M+3 M+4 M+5 M+6 M+7 Machine learning M Optical flow M-7 M-6 M-5 M-4 M-3 M-2 M-1 Save buffer M M+1 M+2 M+3 M+4 M+5 M+6 M+7 Display M-8 M-7 M-6 M-5 M-4 M-3 M-2 M-1

In Table 1, the first group (M-8,...,M-1) in which M-8 is grouped as the first frame, and the second group (M,...M+7) in which M is grouped as the first frame, Indicates the process of processing 2 groups of frames.

Specifically, in the process of collecting frames according to the order described in the frame number (Current Image), a frame number for extracting feature points through a feature point extraction model (machine learning), and a frame for extracting feature points through an optical tracking algorithm (optical flow) Indicates the number, the frame number temporarily stored in the buffer (Save buffer), and the frame number displayed on the linked screen (Display).

For example, if the first frame (M frame) of the second group is collected based on the number of the collected frame, the collected M frame is input into a feature point extraction model (machine learning) and the extracted feature points are stored while the M frame It is also temporarily stored in the buffer. And when collecting M-8 frames, feature points are extracted using an optical algorithm for the first group of M-7 frames temporarily stored in the buffer based on the feature points of the extracted M-8 frames. And the M-8 frame, which is the first frame of the first group, is displayed on the display screen.

Next, the M+1 frames of the second group are collected and temporarily stored in a buffer, and at the same time, feature points are extracted from the M-6 frames of the first group using an optical algorithm based on the extracted feature points of the M-7 frames. Then, the M-7 frame is displayed on the display screen.

By repeating this process, as shown in Table 1, while collecting the second group, feature points are extracted from the first frame of the second group and from the second frame to the last frame of the first group, so that the frames can be displayed without interruption on the display screen.

In this way, the computing device 100 extracts feature points at high speed without interruption by extracting feature points from continuously captured images collected in real time.

In other words, the computing device 100 achieves high accuracy by using a machine learning model that can extract feature points with high accuracy but takes a relatively long time and an optical tracking algorithm that can track at high speed but generates cumulative errors. It is possible to track objects in an image at high speed while extracting feature points.

Figure 8 is an example diagram for explaining a tracking process through matching feature point coordinates in one embodiment of the present invention.

As shown in FIG. 8, the computing device 100 can track an object in an image based on 3D coordinates reconstructed from the Mth color image belonging to the group and the M+1th consecutive color image.

When the computing device 100 extracts the two-dimensional coordinates of a feature point from each color image (shot image), it projects the depth information of the color image or reconstructs it into the three-dimensional coordinates of the feature point through a pinhole camera model.

Accordingly, the computing device 100 tracks the object (feature point coordinates) in the image by performing point-to-point matching on the 3D coordinates of the reconstructed feature point.

Based on the order, the 3D change amount from the 3D coordinates of the reconstructed feature point of the Mth color image to the reconstructed 3D coordinates of the M+1th color image can be calculated.

Additionally, the computing device 100 can apply the 3D change amount to the registered medical image to continuously track the object despite changes in the captured image.

Figure 9 is a hardware configuration diagram of a computing device in one embodiment of the present invention.

Referring to FIG. 9, the image registration module 110, the object tracking module 120, and the learning module 130 are described to execute the operations of the present invention in a computing device 300 operated by at least one processor. Run a program containing instructions.

The hardware of the computing device 300 may include at least one processor 310, memory 320, storage 330, and communication interface 340, and may be connected through a bus. In addition, hardware such as input devices and output devices may be included. The computing device 300 may be equipped with various software, including an operating system capable of running programs.

The processor 310 is a device that controls the operation of the computing device 300, and may be various types of processors 310 that process instructions included in a program, for example, a CPU (Central Processing Unit), MPU ( It may be a Micro Processor Unit (Micro Processor Unit), Micro Controller Unit (MCU), Graphic Processing Unit (GPU), etc. The memory 320 loads the program so that the instructions described to execute the operations of the present invention are processed by the processor 410. The memory 320 may be, for example, read only memory (ROM), random access memory (RAM), etc. The storage 330 stores various data, programs, etc. required to execute the operations of the present invention. The communication interface 340 may be a wired/wireless communication module.

According to the present invention, registration is performed between 3D images in an unconstrained manner using anatomical feature points without using markers and tracking objects, thereby providing fast and accurate registration results and tracking with a minimized amount of calculation without the need for additional equipment such as markers. Data can be secured.

According to the present invention, since the registration of medical images and depth images is performed automatically in real time, it is not affected by the technical skill of the technician for 3D image registration, and thus errors due to differences in skill are minimized to produce registration results with consistently high accuracy. It can be obtained.

According to the present invention, the object can be tracked at high speed while preventing the occurrence of cumulative errors by simultaneously tracking the object using a machine learning model and an optical tracking algorithm based on the anatomical feature points of the object being tracked.

Although the preferred embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims are also possible. falls within the scope of rights.

Claims (16)

1. A method of operating a computing device operated by at least one processor, comprising:
A two-dimensional soft tissue image generated based on a user's medical image is input into a learned feature point extraction model to extract one or more first feature points based on anatomical location, and the first feature point is converted into three-dimensional coordinates based on the medical image. Steps to reconstruct,
A captured image of the user with a depth recognition camera is input to the feature point extraction model to extract one or more second feature points based on the anatomical location, and the depth information of the captured image is applied to extract the second feature point in three dimensions. A step of reconstructing into coordinates, and
Matching the 3D coordinates of the first feature point and the 3D coordinates of the second feature point to match the medical image and the captured image.
A method of operation, including.
In paragraph 1:
The feature point extraction model is,
An operation method that converts an input image into an image based on gradient information and then outputs two-dimensional coordinates of feature points pre-labeled in the image based on gradient information.
In paragraph 1:
The feature point extraction model is,
In the case of an image containing a face, an operation method of selecting and labeling one or more points among the outer end point of the eye, the inner end point of the eye, the corner of the mouth, and the starting point of the nose, where movement according to the face position is minimized, as a feature point.
In paragraph 2,
The step of reconstructing the first feature point into three-dimensional coordinates is,
An operation method of rendering a 3D soft tissue model based on the medical image, generating a 2D soft tissue image projected from the 3D soft tissue model in the coronal plane direction, and inputting it to the feature point extraction model.
In paragraph 4,
The step of reconstructing the first feature point into three-dimensional coordinates is,
An operation method of acquiring 2D coordinates of a first feature point for the 2D soft tissue image and extracting 3D coordinates having a contact point with the 3D soft tissue model using the 2D coordinates of the first feature point as a reference point.
In paragraph 5,
The step of matching the captured images is,
An operating method for performing point-to-point matching between feature points whose labeling matches the 3D coordinates of the first feature point and the 3D coordinates of the second feature point, and calculating coordinate conversion values between the matched feature points. .
In paragraph 6:
The step of matching the captured images is,
An operating method for converting the coordinate system of the medical image into a coordinate system for the captured image by applying the coordinate conversion value to the medical image.
delete delete delete delete delete communication device,
memory, and
At least one processor that executes instructions of a program loaded into the memory,
The above program is
By inputting the user's medical image and the user's captured image taken with a depth recognition camera into a learned feature point extraction model, the coordinates of one or more first feature points and the coordinates of the second feature point, each labeled based on anatomical location, are obtained. , Matching the medical image and the captured image through matching between the coordinates of the first feature point and the coordinates of the second feature point,
If one or more feature point coordinates are extracted for each captured image by selectively using the feature point extraction model and the optical tracking algorithm from the collected continuously captured images, the feature point coordinates are matched between the consecutive captured images, and the amount of change in the matched feature point coordinates A computing device comprising instructions described to calculate and track an object in the captured image.
In paragraph 13:
The above program is,
Rendering a three-dimensional soft tissue model based on the medical image, and generating a two-dimensional soft tissue image projected from the three-dimensional soft tissue model toward the coronal plane,
Obtain the two-dimensional coordinates of the first feature point for the two-dimensional soft tissue image using a feature point extraction model, and extract three-dimensional coordinates having a contact point with the three-dimensional soft tissue model using the two-dimensional coordinates of the first feature point as a reference point. A computing device comprising the described instructions.
In paragraph 14:
The above program is,
Perform point-to-point matching between the 3-dimensional coordinates of the second feature point reconstructed by applying the depth information of the captured image and the feature point whose labeling matches the 3-dimensional coordinates of the first feature point, and matching. A computing device comprising instructions described to calculate coordinate transformation values between feature points.
In paragraph 13:
The above program is,
The sequentially captured images are sequentially grouped based on a preset unit, feature point coordinates are extracted for the first captured image in the group using a learned feature point extraction model, and from the second captured image, feature points extracted from the previous captured image are extracted. A computing device comprising instructions described for extracting feature point coordinates using an optical tracking algorithm based on the coordinates.