KR102140383B1 - Method for microscope based augmented reality navigation, device and computer readable medium for performing the method - Google Patents

Abstract

The microscope-based augmented reality navigation method according to an embodiment of the present invention includes a ratio of a 3D brain model generated from a brain image taken before surgery and a local microscope image to a local area of the brain obtained from a surgical microscope during surgery. Perform rigid body registration, analyze the local microscope image received from the surgical microscope, simulate the physical deformation of the entire 3D brain model according to partial deformation of the local area, and the local microscope image is reflected as augmented reality, A 3D brain model in which simulation results according to a surgical procedure are reflected in real time can be output to a display module provided in a surgical microscope.

Description

Microscope-based augmented reality navigation method, device and recording medium for performing it {METHOD FOR MICROSCOPE BASED AUGMENTED REALITY NAVIGATION, DEVICE AND COMPUTER READABLE MEDIUM FOR PERFORMING THE METHOD}

The present invention relates to a microscope-based augmented reality navigation method, an apparatus for performing the same, and a recording medium, and more specifically, a display module provided in a surgical microscope so that brain deformation occurring during brain surgery can be immediately identified in the middle of surgery. The present invention relates to a microscope-based augmented reality navigation method visualized through augmented reality technology, a device and a recording medium for performing the same.

Augmented Reality (AR) technology has the advantage of high public access due to the increase in realism and immersion by combining real and virtual images. These advantages have spread to the medical field, and research is also being conducted on navigation surgery using AR.

Particularly, brain surgery is one of the surgeries that require precision that can cause disorders due to brain damage to patients with minute errors, and uses augmented reality and navigation systems to accurately locate the lesions.

As a related art, AR-based navigation technology that simply visualizes the result of rigid registration of an existing navigation system through augmented reality (AR) technology using an image taken before surgery (CT/MRI) There is this.

However, the prior art performs only rigid registration of the image before surgery, and despite the fact that it is a target for organs that are deformed by force, such as the brain and liver, the degree of visualizing the main site without considering the deformation of the organ There is a limit that it stops.

In addition, according to the prior art, since the surgical microscope used in brain surgery and the navigation system cannot be used at the same time, there is the inconvenience of alternately viewing the CT/MRI image of the patient matched with the microscope image. ) By using Rigid Registration, an error may occur in the location of the actual brain.

In addition, fluctuations such as cerebrospinal fluid loss, lesion removal, brain beats, etc., which occur during surgery, and brain deformation are not reflected, which makes it difficult to grasp the exact structure of brain tissue that is deformed from time to time during surgery.

Korean Registered Patent No. 10-1190645 Korean Patent Publication No. 10-2017-0134598

One aspect of the present invention estimates the physical deformation of the entire area of the brain with only an image taken by a surgical microscope observing a local area of the brain, and the simulation results of the estimated brain area are augmented reality through a surgical microscope. Provided is a microscope-based augmented reality navigation method that can be visualized, an apparatus for performing the same, and a recording medium.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

The microscope-based augmented reality navigation method according to an embodiment of the present invention includes a ratio of a 3D brain model generated from a brain image taken before surgery and a local microscope image to a local area of the brain obtained from a surgical microscope during surgery. Rigid body registration is performed, and the local microscope image received from the surgical microscope during surgery is analyzed for each frame, and the three-dimensional brain according to partial deformation of the local area according to the positional change of the marker disposed in the local area. Simulates the physical deformation of the entire model and outputs the 3D brain model in which the simulation result according to the surgical process is reflected in real time as the local microscope image is reflected in augmented reality to a display module provided in the surgical microscope. .

Performing the non-rigid registration, after performing the initial matching of the 3D brain model and the local image based on the operating parameters of the surgical microscope, corresponds to the position of each marker included in the local microscope image It may be to extract the 3D coordinates of the 3D brain model to be set as a matching point, and perform precision matching to match the marker and the matching point on a one-to-one basis.

To simulate the physical deformation of the entire 3D brain model, the local microscope image is analyzed frame by frame, and the position of the marker in the previous frame is compared with the position of the marker in the current frame to move for the marker. A vector is calculated, and the matching point pre-matched to the marker is moved according to the motion vector to estimate the physical deformation of the local area, and the physical deformation of the three-dimensional brain model is calculated based on the modified local area. It may be an estimate.

Extracting the brain regions in which the foreground and background are separated from the brain image, generating a high-resolution model representing the extracted brain regions in a mesh form, and generating a plurality of multi-resolution models having different resolutions from the high-resolution models. Further comprising generating, the three-dimensional brain model, if any one of the multi-resolution model is deformed, characterized in that a hybrid model in which the high-resolution model mapped with any one of the multi-resolution model is transformed together Can.

To simulate the physical deformation of the entire 3D brain model, the local microscope image is analyzed frame by frame, and the position of the marker in the previous frame is compared with the position of the marker in the current frame to move for the marker. A vector is calculated, the external force applied to the brain region is estimated based on the calculated at least one motion vector, and the heterogeneous trait regions of the brain region among a plurality of tetrahedral tetrahedra constituting the multi-resolution model Estimating the internal strength according to the constraints preset in the boundary region, and may be to transform each vertex of the tetrahedron constituting the multi-resolution model according to the external force applied to the brain region and the internal force based on the constraint.

Outputting the 3D brain model to the display module provided in the surgical microscope, analyzes operating parameters of the surgical microscope, extracts a region of interest from the local region, renders the region of interest to the high resolution model In other words, the remaining areas excluding the region of interest may be rendered as the multi-resolution model, and the high-resolution model and the three-dimensional brain model rendered as the multi-resolution model may be output to the display module.

Outputting the 3D brain model to the display module provided in the surgical microscope analyzes the operating parameters of the surgical microscope to extract the lesion area formed in the local area, and renders the lesion area to the high resolution model And, the remaining areas other than the lesion area may be rendered by the multi-resolution model, and the high-resolution model and the three-dimensional brain model rendered by the multi-resolution model may be output to the display module.

Outputting the 3D brain model to the display module provided in the surgical microscope extracts a candidate lesion region formed in the local region based on the location information of at least one first marker disposed in the local region, Based on the position of the second marker attached to the surgical tool, a lesion area is extracted from the candidate lesion area, and the deformation of the lesion area is estimated based on a change in the position of the first marker and the second marker in the surgical process. The 3D brain model in which the simulation result is reflected in real time may be output to a display module provided in the surgical microscope.

In addition, in the computer-readable recording medium according to an embodiment of the present invention, a computer program for performing a microscope-based augmented reality navigation method according to the present invention may be recorded.

In addition, the microscope-based augmented reality navigation device according to an embodiment of the present invention, a 3D brain model generated from a brain image taken before surgery, and a local microscope image of a local area of the brain obtained from a surgical microscope during surgery A non-rigid matching unit performing a non-rigid matching of, the local microscope image received from the surgical microscope during surgery is analyzed frame by frame, and partial deformation of the local region according to a change in the position of the marker disposed in the local region The simulation unit for simulating the physical deformation of the entire 3D brain model and the local microscope image are reflected as augmented reality, and the 3D brain model in which the simulation result according to the surgical process is reflected in real time is the surgical microscope It includes an augmented reality control unit output to the display module provided in the.

According to one aspect of the present invention described above, it is possible to provide convenience by providing AR visualization of a surgical microscope with important information related to brain surgery, such as accurate lesion removal depth and area, important blood vessels, and important areas, and shorten the operation time. Can be shortened. In addition, the accuracy of the result of the non-rigid matching can be improved by directly matching the brain, not the skull (scalp). It is possible to provide a navigation system.

1 is a block diagram illustrating a schematic configuration of a microscope-based augmented reality navigation device according to an embodiment of the present invention.
FIG. 2 is a block diagram showing a specific configuration of the non-rigid matching portion of FIG. 1.
FIG. 3 is a diagram illustrating an example of a 3D brain model generated by the hybrid model generator of FIG. 2.
4 is a view showing an example of a specific precision matching performed by the precision matching unit of FIG. 2.
5 to 10 are views illustrating a specific example of a physical model transformation process performed by the simulation unit of FIG. 1.
11 is a flowchart illustrating a schematic flow of a microscope-based augmented reality navigation method according to an embodiment of the present invention.

For a detailed description of the present invention, which will be described later, reference is made to the accompanying drawings that illustrate, by way of example, specific embodiments in which the invention may be practiced. These examples are described in detail enough to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the invention are different, but need not be mutually exclusive. For example, certain shapes, structures, and properties described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with one embodiment. In addition, it should be understood that the location or placement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the invention. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed. In the drawings, similar reference numerals refer to the same or similar functions across various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram showing a schematic configuration of a microscope-based augmented reality navigation device according to an embodiment of the present invention.

The microscope-based augmented reality navigation device 1000 according to the present invention fuses a non-rigid registration technology and a physical simulation technology of brain tissue based on a marker attached to the brain during surgery or a feature point set at a predetermined location in the brain 3D brain shape can be modeled through.

In particular, the microscope-based augmented reality navigation apparatus 1000 according to the present invention estimates the physical deformation of the entire area of the brain only with an image taken by a surgical microscope observing a local area of the brain, and simulates the estimated brain area The results can be visualized with augmented reality through a surgical microscope.

Therefore, the microscope-based augmented reality navigation apparatus 1000 according to the present invention can naturally transform a 3D model of brain tissue in consideration of heterogeneous traits having different physical properties such as lesions and blood vessels, and brain tissue generated during surgery By visualizing the entire deformation in real time through a surgical microscope, the operator can immediately confirm the exact shape of brain tissue during surgery.

Specifically, the microscope-based augmented reality navigation apparatus 1000 according to an embodiment of the present invention may include a rigid body matching unit 100, a simulation unit 200, and an augmented reality control unit 300.

At this time, the microscope-based augmented reality navigation apparatus 1000 according to the present invention may have mobility or be fixed, and may be in the form of a server or an engine, a device, an apparatus, and a terminal. (terminal), user equipment (UE), mobile station (MS), wireless device (wireless device), handheld device (handheld device), etc.

In addition, the microscope-based augmented reality navigation device 1 according to the present invention may be provided in an integrated configuration with a surgical microscope or may be provided separately from a surgical microscope.

Meanwhile, the microscope-based augmented reality navigation apparatus 1000 may be implemented by more components than those shown in FIG. 1, and may be implemented by fewer components. Alternatively, the microscope-based augmented reality navigation device 1000 is a single component by combining at least two components of the non-rigid matching unit 100, the simulation unit 200, and the augmented reality control unit 300 as one component. Can perform multiple functions. Hereinafter, the above-described components will be described in detail.

The non-rigid matching unit 100 is a non-rigid registration of a 3D brain model generated from a brain image taken before surgery and a local microscope image of a local area of the brain obtained from a surgical microscope during surgery (Non-rigid Registration) You can do In this regard, it will be described with reference to FIG. 2 together.

FIG. 2 is a block diagram showing a specific configuration of the non-rigid matching portion 100 of FIG. 1.

As shown, the non-rigid matching unit 100 may include a hybrid model generation unit 110, an initial matching unit 120, and a precision matching unit 130.

The hybrid model generator 110 may generate a 3D brain model from brain images in the form of computed tomography images, magnetic resonance imaging, etc., taken before surgery.

Here, the 3D brain model may be in the form of a hybrid model in which a plurality of multi-resolution models are previously matched to one high-resolution model.

The hybrid model generator 110 may extract a brain region in which the foreground and background are separated from the collected brain image, and generate a high-resolution model in which the extracted brain region is meshed. In addition, the hybrid model generator 110 may generate a plurality of multi-resolution models having different resolutions from the high-resolution models.

That is, a high-resolution model can be generated using a pre-operative imaging image. The high resolution model has a high precision because it represents the brain tissue (brain tissue) in the form of a mesh using a large number of vertices, and is widely used in AR-based navigation surgery because it can express even the smallest parts of the soft body.

However, the high-resolution model has a problem that when the deformation of the brain tissue occurs, the calculation speed for deforming a large number of vertices increases, and the simulation speed is greatly reduced. In addition, the high-resolution model has an empty interior, so it is not possible to take account of regional volumetric deformation, and it is difficult to represent soft tissue composed of heterogeneous traits.

Multiresolution models can be generated from high resolution models. For example, a multi-resolution model can generate a set of tetrahedrons by reducing the number of vertices representing a region of the high-resolution model to the four vertices that make up the tetrahedron.

Since this multi-resolution model represents brain tissue as a set of tetrahedrons, when the brain tissue is deformed, it has an advantage that the simulation speed can be improved by reducing the computation amount for vertex deformation compared to the high-resolution model. In addition, the multi-resolution model represents the brain tissue as a set of tetrahedra and has a continuum characteristic, and may represent deformations due to differences in volume and physical properties of parts of the brain tissue.

The hybrid model generator 110 may perform interpolation between adjacent vertices of a high resolution model and a multiresolution model. For example, the hybrid model generator 110 may perform interpolation between adjacent vertices of a high resolution model and a multiresolution model by applying a linear interpolation method. The hybrid model generator 110 may perform interpolation between the high resolution model and the multiresolution model to resolve the difference in resolution between the high resolution model and the multiresolution model.

The hybrid model generator 110 may generate a 3D brain model in the form of a hybrid model by performing mapping between adjacent vertices of a high resolution model and a multiresolution model. For example, the hybrid model generator 110 may perform coordinate mapping that virtually connects adjacent vertices of the high resolution model and the multiresolution model. In this regard, it will be described with reference to FIG. 3 together.

3 is a diagram illustrating an example of a hybrid model.

As illustrated, the hybrid model generator 110 may generate a hybrid model 1 by interpolating adjacent vertices of the high resolution model 3 and the multiresolution model 5 and performing mapping. In the hybrid model 1, when the multi-resolution model 5 is deformed, the high-resolution model 3 may be deformed together. That is, in order to implement the simulation for transforming the hybrid model 1, the multi-resolution model 5 can be transformed by calculating the deformation physical quantity from the vertices constituting the tetrahedron of the multi-resolution model 5, and such a multi-resolution model The high-resolution model 3 mapped to the vertex of (5) can also be modified together according to the vertex deformation of the multi-resolution model 5. A detailed description in this regard will be described later.

As described above, when the multi-resolution model is deformed by mapping the high-resolution model and the multi-resolution model, the hybrid model generation unit 10 generates a hybrid model that deforms the high-resolution model together, thereby reducing the computational amount for the deformation and simulating speed. In addition to improving, it is possible to reflect heterogeneous traits of tissues, and to visualize a three-dimensional model with high precision by a high-resolution model.

The initial matching unit 120 may perform initial matching of the 3D brain model and the local image based on operating parameters of a surgical microscope. For example, the initial matching unit 120 collects operation parameters indicating the operation of the surgical microscope, such as rotation, movement, and scale conversion of the surgical microscope, so that the local microscope image photographed by the surgical microscope is a 3D brain model. It is possible to roughly estimate which position of the entire area is an image.

The precision matching unit 130 may perform final non-rigid matching of the 3D brain model and the local microscopic image based on the initial matching result performed by the initial matching unit 120.

Specifically, at least one marker for tracking deformation of the brain structure may be disposed in a local area of the brain photographed by a surgical microscope.

The marker is made of a material that is not slipped from the brain surface, such as silicone, and can be placed in a position that does not interfere with surgery because it cannot be directly attached to the brain cortex.

As shown in FIG. 4, the precision matching unit 130 may extract the position of each marker included in the local microscope image, and extract a vertex corresponding to each marker position in the 3D brain model. The precision matching unit 130 may set the extracted vertex as a matching point, and perform precise matching by matching a specific marker and a matching point corresponding to it one-to-one.

On the other hand, the precision matching unit 130 may additionally utilize depth information to match for accurate matching of the augmented reality object, and accordingly, it may express not only accurate matching but also depth sense for actual surgery.

The simulation unit 200 may simulate physical deformation of the 3D brain model.

The local microscope image taken by the surgical microscope shows only a local area of the brain, so the field of view is very narrow. Therefore, the simulation unit 200 may simulate physical deformation of the 3D brain model based on the results of matching the non-rigid cortical brain to provide more accurate information.

As a specific example, the simulation unit 200 may analyze a local microscope image received from a surgical microscope during surgery for each frame to simulate physical deformation of a 3D brain model. In this regard, it will be described with reference to FIGS. 5 to 7 together.

5 to 7 are views for explaining a specific example of a physical model deformation process according to a change in the position of the marker.

First, as illustrated in FIG. 5, the simulation unit 200 may collect local microscope images received from a surgical microscope during surgery for each frame. In the illustrated embodiment, it can be seen that the position of the marker disposed in the brain tissue in the previous frame (t Frame) is changed as the surgical tool applies external force to the brain tissue in the current frame (t+1 Frame).

In this case, the simulation unit 200 analyzes the local microscope image for each neighboring frame as shown in FIG. 6, and the position of the marker in the previous frame (t Frame) and the marker in the current frame (t+1 Frame) You can compare the locations. The simulation unit 200 may calculate a motion vector representing a change in the position of the marker for each marker.

Thereafter, as illustrated in FIG. 7, the simulation unit 200 may estimate a physical deformation of the local area by moving according to a motion vector calculating a matching point pre-matched for each marker. Finally, the simulation unit 200 may estimate the physical deformation of the entire 3D brain model based on the modified local area. That is, the simulation unit 200 simulates the deformed local area by moving each matching point as much as the motion vector calculated for each matching point, and connects the deformed local area and the neighboring area among the non-deformed areas to the 3D brain. You can estimate the overall deformation of the model.

As another example, the simulation unit 200 may simulate the physical deformation of the 3D brain model by estimating the external force and internal stress applied to the local area in the course of surgery, and analyzing it based on this. In this regard, it will be described with reference to FIGS. 8 and 9 together.

8 is an example of a heterogeneous trait region of brain tissue set by the simulation unit, and FIG. 9 is a diagram showing an example of a virtual spring generated by the simulation unit.

As illustrated, the simulation unit 200 may set a heterogeneous trait region in brain tissue represented by the hybrid model, and generate a virtual spring connecting the heterogeneous trait regions.

Human organs can consist of a collection of heterogeneous traits with different properties, such as general tissue, lesions, and blood vessels. Therefore, when a 3D model deformation simulation is performed on the assumption that the physical properties of brain tissue are one, errors may occur and unnatural results may occur.

Accordingly, the simulation unit 200 may generate a virtual spring that becomes a constraint when a multi-resolution model is deformed on a tetrahedron of a multi-resolution model representing a boundary region between heterogeneous trait regions of brain tissue.

First, the simulation unit 200 may set a heterogeneous trait region of brain tissue in a hybrid model, and set a boundary region between heterogeneous trait regions.

For example, referring to FIG. 8, the simulation unit 200 may set the hybrid model representing the brain as the first trait region 1b of normal tissue and the second trait region 1a of tumor tissue, and the first The region between the boundary of the trait region 1b and the boundary of the second trait region 1a can be set as the boundary region 2 between the heterogeneous trait regions.

Here, when a modification to the first trait region 1b or the second trait region 1a is made, the tetrahedron of the multi-resolution model representing the first trait region 1b or the second trait region 1a is constructed. The deformation of the vertex is simulated, and the physical quantity of the vertex deformation can be calculated by applying different constraints based on physical properties representing characteristics of normal tissue or tumor tissue, respectively.

The simulation unit 200 may obtain coordinates of each vertex constituting the tetrahedron of the multi-resolution model representing the boundary region.

That is, as shown, the boundary region 2 may be represented by a set of tetrahedrons 5a of the multi-resolution model. The simulation unit 200 may acquire coordinates of each vertex constituting the tetrahedron 5a representing the boundary region 2.

Meanwhile, the simulation unit 200 may generate a virtual spring for connecting the heterogeneous trait regions in the boundary region between the heterogeneous trait regions. The virtual spring may be defined as a line connecting each vertex constituting the tetrahedron of the multi-resolution model and the inner center point of the tetrahedron.

Referring to FIG. 9, the tetrahedron of the multi-resolution model representing the boundary region may be composed of four vertices. The simulation unit 200 may generate the virtual spring 7 by connecting these four vertex coordinates (p ₁ , p ₂ , p ₃ , p ₄ ) and the internal center point coordinates (p ₀ ). Here, the coordinates of the inner center point (p ₀ ) of the multi-resolution model tetrahedron may be calculated as in Equation 1 below.

In Equation 1, p ₀ denotes the center point coordinates of the multi-resolution model tetrahedron, and p _i denotes vertex coordinates constituting the multi-resolution model tetrahedron.

The simulation unit 200 may set the elastic modulus of the virtual spring based on different physical properties between heterogeneous trait regions. The simulation unit 200 may set the elastic modulus of the virtual spring using the property values representing the characteristics of the first trait region 1b and the second trait region 1a.

To this end, the simulation unit 200 may set constraints of the boundary region 2. The simulation unit 200 may calculate constraints applied when the tetrahedron 5a representing the boundary region 2 is deformed using the virtual spring as shown in Equations 2 and 3 below.

는 정점의 변형 방향, k_hetero는 가상 스프링의 탄성계수를 나타낸다. 수학식 3에서 l_i는 가상 스프링을 구성하는 두 정점 간의 초기 거리를 나타낸다.In Equation 2, w _i represents the weight of the vertex p _i , and C _spring is a distance difference between two vertices constituting the virtual spring, and can be calculated as in Equation 3. Also, in Equation 2

Is the deformation direction of the vertex, k _hetero is the elastic modulus of the virtual spring. In Equation 3, l _i represents the initial distance between two vertices constituting the virtual spring.

In this way, the simulation unit 200 may set a heterogeneous trait region in brain tissue represented by the hybrid model, and generate a virtual spring connecting the heterogeneous trait regions.

Thereafter, the simulation unit 200 may estimate deformation of the 3D brain model using the set heterogeneous trait region and the generated virtual spring. In this regard, it will be described with reference to FIG. 10 together.

10 is a diagram illustrating a specific example of a physical simulation method of a hybrid model performed by a simulation unit.

Specifically, the simulation unit 200 may deform the hybrid model by reflecting external forces applied to brain tissue and internal forces based on constraints.

In order to implement the simulation for transforming the hybrid model 1 as described above, the multi-resolution model 5 can be transformed by calculating the deformation physical quantity from the vertices constituting the tetrahedron of the multi-resolution model 5, The high-resolution model 3 mapped to the vertex of the resolution model 5 can also be modified together according to the vertex deformation of the multi-resolution model 5.

Therefore, the simulation unit 200 may change the vertex of the multi-resolution model 5, and in this case, in the case of the multi-resolution model showing the boundary region between heterogeneous trait regions, the constraint condition according to Equation 2 may be reflected.

The simulation unit 200 may use a PBD (Position Based Dynamics) physical model as a physical model for performing the vertex transformation.

At this time, the simulation unit 200 may perform the four steps repeatedly to deform the vertex of the multi-resolution model 5 to realize a physical simulation of the hybrid model 1 representing brain tissue.

First, the simulation unit 200 may calculate the new position of the vertex by reflecting the speed and external force for the unit time based on the current position of each vertex constituting the tetrahedron of the multi-resolution model 5. Specifically, the simulation unit 200 calculates a new velocity (v _i ^new ) in consideration of the external force (f) applied to the brain tissue and the internal damping and mass (m _i ) at the current velocity (v _i ) of the vertex. Can.

Here, the magnitude and direction of the external force can be obtained from a motion vector indicating a change in the position of the marker calculated by comparing the position of the marker in the previous frame (t Frame) with the position of the marker in the current frame (t+1 Frame). have.

In addition, the simulation unit 200 reflects the new speed (v _i ^new ), which is speed information for the unit time (△ t), based on the current position (p _i ) of the vertex, and sets the new position (p _i ^new ) of the vertex. Can be calculated. This can be expressed as Equations 4 to 6 below.

The simulation unit 200 may modify the final position p _i ^fin by applying the constraint C to the new position p _i ^new . Here, in the case of a multi-resolution model showing a boundary region between heterogeneous trait regions, constraints according to Equation 2 based on a virtual spring may be applied. This can be expressed as Equations 7 and 8 below.

Finally, the simulation unit 200 may prevent a phenomenon in which the modified hybrid model 1 penetrates by performing collision processing.

As described above, the simulation unit 200 may provide a simulation result in consideration of the external force and the internal force of the soft body by performing the transformation of the hybrid model through the vertex transformation based on the PBD physical model. In particular, when the boundary region between heterogeneous traits is modified, the deformation of the heterogeneous trait region can be naturally expressed by applying a constraint based on a virtual spring.

The augmented reality control unit 300 may control the final simulated 3D brain model to be output through a display module provided in the surgical microscope.

The augmented reality control unit 300 may use a non-rigid matching result to output an image in which a local microscope image is reflected as an augmented reality in a 3D brain model through a display module.

Subsequently, the augmented reality control unit 300 reflects the simulation result according to the physical change of the 3D brain model generated in the surgical process in real time.

Meanwhile, the augmented reality control unit 300 may dynamically change the resolution of the 3D brain model according to the operating state of the surgical microscope collected during the surgical procedure.

For example, the augmented reality control unit 300 may extract an area of interest from a local area by analyzing operating parameters of a surgical microscope. That is, the augmented reality controller 300 may collect signals or information related to the operation of the surgical microscope, and determine which part of the brain area the operator is currently interested in (gaze). .

The augmented reality controller 300 models the brain region corresponding to the extracted region of interest as a high-resolution model having the highest resolution among the hybrid models, and sets the remaining regions of the brain model excluding the extracted region of interest to have a lower resolution than the high-resolution model. It can be rendered into the first multi-resolution model. In this process, the augmented reality controller 300 may render a region not included in the region of interest and a region related to the region of interest (blood vessels, etc.) as a second multi-resolution model having a higher resolution than the first multi-resolution model.

Thereafter, the augmented reality control unit 300 may output the high-resolution model and the 3D brain model rendered as the first and second multi-resolution models in real time through a display module provided in the surgical microscope. Accordingly, the microscope-based augmented reality navigation apparatus 1000 according to the present invention has an effect of accurately providing modeling information on an area required for an operator while minimizing a processing speed required for simulation.

As another example, the augmented reality control unit 300 may accurately extract a lesion area deformed according to a surgical process among brain areas using location information of a marker attached to a surgical tool.

Specifically, the augmented reality control unit 300 may extract a candidate lesion region formed in the local region based on the location information of at least one first marker disposed in the local region included in the local microscope image. As described above, since the marker attached to the brain cortex is attached to a position in the brain area adjacent to the lesion and not directly affecting the lesion, the augmented reality control unit 300 of the first marker fixedly placed in the local area The location can be used to estimate the approximate lesion area.

Thereafter, the augmented reality control unit 300 may analyze the local microscope image collected in the surgical process, and collect the position of the second marker, which is a marker attached to the surgical tool, by frame. Since the surgical tool directly accesses the lesion, the augmented reality control unit 300 generates a predetermined radius based on the position of the second marker attached to the surgical tool as a reference point, and finalizes the region overlapping with the previously estimated candidate lesion region. It can be estimated as the lesion area. Thereafter, the augmented reality control unit 300 may output a 3D brain model in which the deformation of the estimated lesion region is reflected in real time based on the positional changes of the first marker and the second marker to a display module provided in the surgical microscope.

11 is a flowchart illustrating a schematic flow of a microscope-based augmented reality navigation method according to an embodiment of the present invention.

The microscope-based augmented reality navigation method according to an embodiment of the present invention may be performed by the microscope-based augmented reality navigation device 1000 of FIG. 1. To this end, the microscope-based augmented reality navigation apparatus 1000 according to the present invention may be pre-installed with software (application) for performing each step constituting the microscope-based augmented reality navigation method described later.

First, the microscope-based augmented reality navigation apparatus 1000 performs non-rigid matching of a 3D brain model generated from a brain image taken before surgery and a local microscope image of a local area of the brain obtained from a surgical microscope during surgery. It can be done (910).

The microscope-based augmented reality navigation apparatus 1000 may generate a high-resolution model represented in a mesh form from brain images taken before surgery, and may generate a plurality of multi-resolution models having different resolutions from the generated high-resolution model, A hybrid model composed of a plurality of multi-resolution models mapped to a model may be set as a 3D brain model. The microscope-based augmented reality navigation device 1000 may perform non-rigid matching with a local microscope image photographed from a surgical microscope on the generated 3D brain model.

The microscope-based augmented reality navigation apparatus 1000 may analyze a local microscope image received from a surgical microscope during surgery to simulate physical deformation of the entire 3D brain model according to partial deformation of the local area (920). .

Specifically, the microscope-based augmented reality navigation apparatus 1000 calculates a motion vector representing a change in the position of a marker included in a local microscope image collected for each frame, and partially transforms a local area of the brain according to the calculated motion vector Can be estimated, and based on this, physical deformation of the entire 3D brain model can be simulated. Since detailed contents related to this have been described above, repeated description will be omitted.

Thereafter, the microscope-based augmented reality navigation apparatus 1000 may output a 3D brain model in which a simulation result according to a surgical process is reflected in real time while a local microscope image is reflected as augmented reality to a display module provided in the surgical microscope. Yes (930).

Such a technique for providing a microscope-based augmented reality navigation method may be implemented as an application or implemented in the form of program instructions that can be executed through various computer components to be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, or the like alone or in combination.

The program instructions recorded on the computer-readable recording medium are specially designed and configured for the present invention, and may be known and available to those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include not only machine language codes produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although described above with reference to embodiments, those skilled in the art understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. Will be able to.

Although described above with reference to embodiments, those skilled in the art understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. Will be able to.

100: non-rigid mating part
200: simulation unit
300: augmented reality control

Claims (10)

Non-rigid matching of a 3D brain model generated from a brain image taken before surgery and a local microscopic image of a local area of the brain obtained from a surgical microscope during surgery is performed.
During surgery, the local microscope image received from the surgical microscope is analyzed frame by frame to physically transform the entire 3D brain model according to the partial deformation of the local area according to the positional change of the marker disposed in the local area. To simulate
A microscope-based augmented reality navigation method for outputting the 3D brain model in which a simulation result according to a surgical procedure is reflected in real time to the display module provided in the surgical microscope while the local microscope image is reflected in the augmented reality.
The method of claim 1, wherein performing the non-rigid matching,
After performing the initial matching of the 3D brain model and the local microscope image based on the operating parameters of the surgical microscope, 3 of the 3D brain model corresponding to the position of each marker included in the local microscope image A method of extracting dimensional coordinates, setting them as a matching point, and performing precision matching to match the marker and the matching point on a one-to-one basis, a microscope-based augmented reality navigation method.
According to claim 2, Simulating the physical deformation of the entire three-dimensional brain model,
The local microscope image is analyzed frame by frame to compare the position of the marker in the previous frame and the position of the marker in the current frame to calculate a motion vector for the marker, and to match the matching point previously matched to the marker. A method for estimating a physical deformation of the local area by moving according to the motion vector, and estimating a physical deformation of the three-dimensional brain model based on the modified local area.
According to claim 1,
Extracting the brain regions in which the foreground and background are separated from the brain image, generating a high-resolution model representing the extracted brain regions in a mesh form, and generating a plurality of multi-resolution models having different resolutions from the high-resolution models. Further comprising generating,
The 3D brain model is a microscope-based augmented reality navigation method, characterized in that, when any one of the multi-resolution models is modified, the high-resolution model mapped with the one multi-resolution model is transformed together. .
According to claim 4, Simulating the physical deformation of the entire three-dimensional brain model,
The local microscope image is analyzed frame by frame to compare the position of the marker in the previous frame and the position of the marker in the current frame to calculate a motion vector for the marker, and based on the calculated at least one motion vector To estimate the external force applied to the brain region,
Estimation of a history according to a preset constraint in a boundary region between heterogeneous trait regions of the brain region among a plurality of tetrahedron tetrahedra constituting the multiresolution model,
A microscope-based augmented reality navigation method for transforming each vertex of the tetrahedron constituting the multi-resolution model according to the external force applied to the brain region and the internal force based on the constraint.
According to claim 4, Outputting the three-dimensional brain model to the display module provided in the surgical microscope,
The operating parameters of the surgical microscope are analyzed to extract a region of interest from the local region, render the region of interest to the high-resolution model, render the remaining regions excluding the region of interest to the multi-resolution model, and the high-resolution model And outputting the 3D brain model rendered as the multi-resolution model to the display module.
According to claim 4, Outputting the three-dimensional brain model to the display module provided in the surgical microscope,
By analyzing the operating parameters of the surgical microscope, a lesion region formed in the local region is extracted, the lesion region is rendered into the high resolution model, and the remaining regions excluding the lesion region are rendered into the multiresolution model, and the high resolution And outputting the 3D brain model rendered as a model and the multi-resolution model to the display module.
According to claim 4, Outputting the three-dimensional brain model to the display module provided in the surgical microscope,
A candidate lesion area formed in the local area is extracted based on the location information of at least one first marker disposed in the local area, and a lesion area from the candidate lesion area based on the position of the second marker attached to the surgical tool Extracting, and estimating the deformation of the lesion area based on the positional changes of the first marker and the second marker, and providing the 3D brain model in which the simulation results according to the surgical process are reflected in real time in the surgical microscope A microscope-based augmented reality navigation method that outputs to a display module.
A computer-readable recording medium having a computer program recorded thereon for performing the microscope-based augmented reality navigation method according to any one of claims 1 to 8.
A non-rigid matching unit that performs non-rigid matching of a local microscopic image on a local area of the brain obtained from a surgical microscope during surgery and a 3D brain model generated from a brain image taken before surgery;
During surgery, the local microscope image received from the surgical microscope is analyzed frame by frame to physically transform the entire 3D brain model according to the partial deformation of the local area according to the positional change of the marker disposed in the local area. Simulation unit for simulating; And
A microscope-based augmentation including an augmented reality control unit that outputs the 3D brain model in which the simulation result according to the surgical process is reflected in real time to the display module provided in the surgical microscope while the local microscope image is reflected in the augmented reality. Reality navigation device.

Images