KR20230135448A - Apparatus and method for arthroscopy based on movement of bones in joint - Google Patents

Abstract

The joint surgery navigation device according to an embodiment of the present disclosure includes a storage unit that stores a joint 3D model in which soft tissue and hard tissue are distinguished based on a 3D image of the patient and different physical properties are given to the soft tissue and hard tissue, a processor, and It is electrically connected to the processor and includes a memory in which at least one code executed by the processor is stored, and when the memory is executed through the processor, the processor determines an external force applied to the patient and determines the joint condition based on the external force. Calculate the deformation of the soft tissue in the 3D model, change the position of the hard tissue in the 3D model of the joint based on the deformation of the soft tissue in the 3D model, and change the position of the arthroscope for taking images of the patient's joint area. Code that causes at least a portion of the three-dimensional model to be displayed visually along with images taken from the arthroscope may be stored.

Description

Arthroscopic navigation device and method reflecting the movement of bones inside a joint {APPARATUS AND METHOD FOR ARTHROSCOPY BASED ON MOVEMENT OF BONES IN JOINT}

The present disclosure relates to an arthroscopic navigation device and method, and more specifically, to a navigation device and method that accurately reflects the movement state of bones inside a joint in real time during joint surgery.

Conventionally, in the case of joint surgery including knee joints, shoulder joints, wrist joints, ankle joints, etc., medical staff insert an arthroscope through a small hole (portal) less than 1 cm and view the damaged area in the joint through a monitor. Arthroscopic surgery can be performed to perform both diagnosis and treatment.

In the case of joints, multiple bones are connected to each other in a complex organic way to generate joint movement, and unlike other parts of the human body, it is difficult to look inside properly even when incised, and due to limitations in the surgical space using a general surgical mass, examination using an arthroscope is performed. and treatment is performed frequently.

However, even in the case of surgery using arthroscopy, there are cases where arthroscopic surgery is performed after the bone is pulled within a certain range to secure the surgical space. In order to safely insert arthroscopic instruments into the joint, the joint is performed using a radioscopic device. Arthroscopic surgery can be performed while understanding the internal structure and movement of the arthroscope.

In this case, the use of radioscopic devices causes radiation exposure problems for medical staff and patients, and to solve this problem, the position of surgical tools or arthroscopes must be determined using surgical navigation technology based on optical or magnetic sensors or electromagnetic (electromagnetic) technology as in prior literature 1. ) Surgical navigation technologies based on sensors and augmented reality also exist.

Conventional arthroscopic surgery navigation technologies, including Prior Literature 1, reconstruct patient images taken before surgery in three dimensions to guide arthroscopic surgery (navigation), then display them by registering them with the position of the arthroscope and a coordinate system. In this case, since it does not reflect the actual movement of the bone due to traction, there is a problem in that there is a discrepancy between the image shown on the surgical scope and the simulated image of the surgical guidance device. In particular, in the case of a joint in which multiple bones move in a complex manner, the portal through which the arthroscope is inserted must be selected considering the positions of the moved bones. In this case, there is a risk that medical staff may select the wrong portal.

In addition, the prior art has a problem in that when the joint is moved during surgery to ensure a smooth view of the arthroscope, the actual surgical environment cannot be further reflected, the accuracy decreases, and the more the movement is repeated, the greater the decrease in accuracy.

Prior Art 1: Zemirline A et al., “Augmented reality-based navigation system for wrist arthroscopy: feasibility”, Journal Wrist Surgery, Vol. 2, No. 4, pp294-298, November 2013

An embodiment of the present disclosure provides a navigation device and method that reflects the movement state of bones inside a joint in real time during joint surgery.

Embodiments of the present disclosure provide a navigation device and method that accurately reflects the movement state of bones inside a joint due to traction.

Embodiments of the present disclosure provide a navigation device and method that takes into account the force that external force due to traction acts on the bone through muscle.

Embodiments of the present disclosure provide a navigation device and method capable of selecting an appropriate portal location to prevent damage inside the joint.

The joint surgery navigation method according to an embodiment of the present disclosure is a method in which at least part of each step is performed by a processor, in which soft tissue and hard tissue are distinguished based on a 3D image of the patient, and soft tissue and hard tissue have different physical properties. Step of loading the given joint 3D model, determining the external force applied to the patient, calculating the deformation of the soft tissue of the joint 3D model based on the external force, joint 3 based on the deformation of the soft tissue of the joint 3D model It may include changing the position of the hard tissue of the dimensional model and visually displaying at least a portion of the 3-dimensional joint model based on the position of the arthroscope.

The joint surgery navigation device according to an embodiment of the present disclosure includes a storage unit that stores a joint 3D model in which soft tissue and hard tissue are distinguished based on a 3D image of the patient and different physical properties are given to the soft tissue and hard tissue, a processor, and It is electrically connected to the processor and includes a memory in which at least one code executed by the processor is stored, and when the memory is executed through the processor, the processor determines an external force applied to the patient and determines the joint condition based on the external force. Calculate the deformation of the soft tissue in the 3D model, change the position of the hard tissue in the 3D model of the joint based on the deformation of the soft tissue in the 3D model, and change the position of the arthroscope for taking images of the patient's joint area. Code that causes at least a portion of the three-dimensional model to be displayed visually along with images taken from the arthroscope may be stored.

The joint surgery navigation device and method according to an embodiment of the present disclosure can reflect the movement state of bones inside the joint in real time during joint surgery.

The joint surgery navigation device and method according to an embodiment of the present disclosure can accurately reflect the state of bone movement due to external force.

The joint surgery navigation device and method according to an embodiment of the present disclosure can reduce the risk of inappropriate portal positioning for joint surgery.

Figure 1 is a diagram showing the actual range of bone movement during wrist joint surgery.
Figure 2 is a block diagram showing the usage environment of the joint surgery navigation device according to an embodiment of the present disclosure.
Figure 3 is a block diagram showing the configuration of a joint surgery navigation device according to an embodiment of the present disclosure.
Figure 4 is a diagram explaining the bone structure of the wrist joint of the present disclosure.
Figure 5 is a flowchart explaining a joint surgery navigation method according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating a three-dimensional joint model according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating links of a joint 3D model according to an embodiment of the present disclosure.
FIG. 8 is a diagram illustrating feature points of a joint 3D model according to an embodiment of the present disclosure.
9 to 12 are diagrams illustrating a method of configuring links of a joint 3D model according to an embodiment of the present disclosure.
Figure 13 is a diagram showing the results of displaying a three-dimensional joint model in augmented reality reflecting bone movement according to an embodiment of the present disclosure.
FIG. 14 is a diagram showing experimental errors of bone movement simulation reflecting traction external force in the navigation device and method according to an embodiment of the present disclosure.
15 and 16 are diagrams illustrating a display interface of a navigation device and method according to an embodiment of the present disclosure.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numbers regardless of reference numerals, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Figure 1 shows the actual lunate bone (Figure 1 (a)), scaphoid bone (Figure 1 (b)), mammillary bone (Figure 1 (c)), and cuneiform bone (Figure 1 (d)) included in the wrist joint during wrist joint surgery. )) shows the result of movement due to traction. It can be seen that the difference between the pre-operative position (Neutral) and the moved position of the bone due to traction (Real traction) is quite large, which means that in the case of arthroscopic surgery that requires fine movement of the surgical tool, the changed position of the bone You can confirm its importance.

An environment for driving a joint surgery navigation device according to an embodiment of the present disclosure will be described with reference to FIG. 2 .

The joint surgery navigation device 100 according to an embodiment of the present disclosure includes an arthroscope 300 with optical or magnetic markers 310 and 1200 attached thereto, and a position tracking device 200 for tracking the position of the arthroscope 300. , It can be driven in conjunction with the image display device 400 that displays the camera image of the arthroscope 300.

The position tracking device 200 may be an optical camera-based position tracking device 200 when tracking the optical marker of the arthroscope 300 or the patient 1000, and may be based on a sensor capable of detecting the distance of an object. You can. For example, it may include an optical camera based on a depth sensor such as an RGB-D sensor. Alternatively, it may be a stereo camera. The location tracking device 200 may be an optical tracker, and in this case, the markers 310 and 1200 may be optically recognizable (retro-reflective) markers in the optical tracker.

The optical tracker may be an active optical tracker or a passive optical tracker based on optics such as infrared. Optical trackers can be monocular or stereo-based cameras. When using passive optical tracking technology, the optical tracker may include an infrared illumination LED.

The arthroscope 300 may include a gyro sensor or an inertial sensor at the distal end or part of the arthroscope, and the communication module 110 of the position tracking device 200 or the joint surgery navigation device 100 is a gyro sensor of the arthroscope 300. , the posture of the arthroscope 300 can be determined based on the data measured by the inertial sensor. The joint surgery navigation device 100 displays the corresponding part of the joint 3D model in consideration of the field of view (FOV) of the arthroscope 300 camera according to the change in posture of the arthroscope 300. ) can be displayed.

The image display device 400 includes all display devices capable of displaying images. The image display device 400 may be a general computer monitor, augmented reality glasses worn by medical staff, or beam projection. In the case of beam projection, the joint surgery navigation device 100 may project a three-dimensional joint model based on the patient's position and posture.

The medical staff may select a portal for inserting the arthroscope 300 into the joint of the patient 1000 by considering the simulation image of the joint surgery navigation device 100. In this case, the joint surgery navigation device 100 confirms the position of the arthroscope 300 and, before inserting the arthroscope 300 inside the patient, displays a 3D model of the joint in augmented reality on the external image of the surgical site as shown in FIG. 15. It can be displayed. Therefore, the medical staff can safely select the portal by considering bone movement due to external forces such as traction.

In one embodiment, the joint surgery navigation device 100 determines the position of a portal suitable for arthroscopic insertion as shown in FIG. 15 based on a learning model based on machine learning during joint surgery and displays a three-dimensional view of the joint on the image display device 400. It can be displayed together with the model. The joint surgery navigation device 100 confirms the position and viewing angle of the external arthroscope 310 with respect to the patient, generates a planar image of the joint 3D model at the corresponding viewing angle, and converts the planar image of the joint 3D model into a learning model. You can determine the location of the portal by entering . The learning model may be a deep learning-based learning model trained with training data labeling a planar image of a joint with the location of the portal. The planar image is a virtual image created by setting a ray path in the direction of looking at the three-dimensional joint model from the position of the X-ray image or arthroscope 310 and accumulating voxels of each ray path. virtual) may be a flat image.

Machine learning-based learning models include CNN or R-CNN (Region based CNN), C-RNN (Convolutional Recursive Neural Network), Fast R-CNN, Faster R-CNN, and R-FCN (Region based Fully Convolutional Network). ), YOLO (You Only Look Once), or SSD (Single Shot Multibox Detector) structures.

The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory.

The joint surgery navigation device 100 loads a 3D model of the joint in which soft tissue and hard tissue are segmented and given different physical properties, and displays a marker 1200 due to external force applied to the patient 100 by traction, etc. ) can be provided from the position tracking device 200, and based on the changed position of the marker 1200, the deformation of the soft tissue and the change in the position of the hard tissue due to the deformation of the soft tissue can be modeled. Alternatively, the joint surgery navigation device 100 may measure the positional movement change and traction force of the traction unit connected to the patient 1000 in the traction device to model the deformation of the soft tissue and the change in the position of the hard tissue due to the deformation of the soft tissue. Changes in the position of hard tissue can be calculated based on links between a plurality of bones included in the hard tissue, which will be explained below. The joint surgery navigation device 100 may visually display at least a portion of the 3D model of the joint in which the position of the hard tissue has changed on the image display device 400 in real time.

The joint surgery navigation device 100 registers the coordinate system of the joint 3D model, the patient's coordinate system, and the arthroscope's coordinate system, and displays at least a portion of the joint 3D model on the image display device 400 using the arthroscope 300. It can be displayed overlapping with the image acquired from or overlapping with the patient's image. Alternatively, at least part of the three-dimensional joint model may be displayed as a separate image without overlapping with the image acquired by the arthroscope 300 or the patient's image.

In one embodiment, the joint surgery navigation device 100 or the arthroscope control device performs hand-eye calibration between the marker 310 of the arthroscope 300 and the tooltip of the arthroscope 300. After performing, the location of the camera located at the distal end can be specified.

The configuration of the joint surgery navigation device 100 according to an embodiment of the present disclosure will be described with reference to FIG. 3 .

The joint surgery navigation device 100 includes a position tracking device 200, an arthroscope 300, and/or an image display device 400, and a communication module 100 for data transmission. The communication module 100 may include a wireless communication unit or a wired communication unit.

The wireless communication unit may include at least one of a mobile communication module, a wireless Internet module, a short-range communication module, and a location information module.

The mobile communication module transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network built according to LTE (Long Term Evolution), a communication method for mobile communication.

The wireless Internet module is a module for wireless Internet access and can be built into or external to the joint surgery navigation device 100, and can be connected to WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), and Wi-Fi (Wireless Fidelity) Direct. , DLNA (Digital Living Network Alliance), etc. may be used.

The short-range communication module is a module for transmitting and receiving data through short-range communication, including Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, and NFC (Near Field). Communication), etc. can be used.

The communication module 100 includes a cable that transmits image data to the display unit 120 or the image display device 400 including a monitor or receives image data captured by the arthroscope 300 and its connection interface. It can be included.

When the joint surgery navigation device 100 is integrated and implemented as an arthroscopic surgery device that controls the mechanical and electronic assembly of the arthroscope 300, the processor 130 sends a signal to control each component of the arthroscope 300. It may occur, and the image captured by the arthroscope 300 camera may be pre-processed and displayed on the display unit 120. In this case, components of a conventional arthroscopic surgical device may be additionally included, and techniques known to those skilled in the art may be used, so detailed description is omitted.

The processor 130 of the joint surgery navigation device 100 is a 3D image reconstructed by imaging the patient in a 3D diagnostic imaging device such as computed tomography (CT) imaging or magnetic resonance imaging (MRI). An image may be provided through the communication module 110, or an image obtained by segmenting and labeling soft tissue and hard tissue from a 3D image of the patient may be provided and stored in the storage unit 150.

The joint surgery navigation device 100 receives data classified by tissue based on various conventional classification techniques such as pixel (or voxel) intensity, image feature, and contour of soft tissue and hard tissue 3D images. , property information can be set for each organization, or 3D images can be provided, classification technology can be applied, and property information can be set for each organization.

As can be seen in FIG. 6, the joint 3D model of the present specification may be data in which each tissue (soft tissue, hard tissue) is distinguished and labeled in a 3D image. In addition, the joint 3D model can specify different physical property information for each tissue (at least distinguishing between hard and soft tissues), and the physical property information includes mass, inertia tensor, and Poisson ratio. ), Young's modulus, and stiffness, etc. may be material information. The joint surgery navigation device 100 performs a simulation based on 3D finite element analysis (FEA) by applying external forces such as traction force applied to the patient to a 3D joint model in which physical property information is set for each tissue. The change in position of the hard tissue can be determined by calculating the deformation and conducting finite element analysis simulation of the force transmitted to the hard tissue connected to the soft tissue due to the deformation of the soft tissue. At this time, the joint surgery navigation device 100 may determine a change in the position of the hard tissue based on a link as shown in FIG. 7 established based on the movement relationship between a plurality of bones included in the hard tissue.

Hard tissues may include bones, etc., and soft tissues may include muscles, ligaments, etc. In the case of a joint, the hard tissue may include a plurality of bones that make up the joint, and in the case of the wrist joint, the cuneiform bone, mammillary bone, hamate bone, capitate bone, deltoid bone, lunate bone, scaphoid bone, and/or major bone in FIG. May include at least part of the rhomboid bone.

In one embodiment, the joint surgery navigation device 100 may include an input unit or output unit for user input.

The input unit includes a user interface (UI: User Interface) including a microphone and a touch interface for receiving information from the user, and the user interface may include a mouse, a keyboard, as well as mechanical and electronic interfaces implemented in the device. As long as the user's command can be input, the method and form are not particularly limited. The electronic interface includes a display capable of touch input.

The output unit is for delivering information to the user by displaying the output of the joint surgery navigation device 100 to the outside, and may include a display, LED, speaker, etc. for expressing visual output, auditory output, or tactile output. .

The joint surgery navigation device 100 may include a peripheral device interface unit for data transmission with various types of connected external devices, a memory card port, and an external device I/O (Input/Output) port. ), etc. may be included.

A joint surgery navigation method of a joint surgery navigation device according to an embodiment of the present disclosure will be described with reference to FIG. 5 .

The joint surgery navigation device 100 loads the joint 3D model from a temporary storage device such as memory or a non-volatile storage device such as a solid state drive (SSD) or hard disk drive (HDD), or downloads the joint 3D model from an external device through a communication interface. It can be provided (S110). Alternatively, the joint surgery navigation device 100 may receive a 3D image of the patient from an external device and label the soft tissue and hard tissue separately as shown in FIG. 6 through image processing. The joint surgery navigation device can set different physical property information for soft and hard tissues in a three-dimensional joint model in which soft and hard tissues are distinguished.

The joint surgery navigation device 100 recognizes, in the position tracking device 200, a change in the position of the marker 1200 attached to the patient's skin or the marker attached to the traction unit of the traction device for towing the joint 1100 or the traction device Based on the traction force measured in, the size and direction of the external force are determined, the external force is applied to the soft tissue of the joint 3D model, and a 3D finite element analysis simulation is performed to calculate the deformation of the soft tissue (S120). The degree of movement due to traction of the distal hard tissue of the hard tissue link path along the hard tissue link path can be calculated, and the movement and/or position of the hard tissues connected to the link included in the same link path can be changed (S130).

For example, in the case of Figure 7 (a), when the traction unit of the traction device pulls the patient's skin upward from the outside of the papillary bone, the joint surgery navigation device 100 exerts an external force applied to the skin and soft tissue beneath the skin. is determined through finite element analysis simulation, and the extent to which the papillary bone connected to the soft tissue moves upward is also determined through finite element analysis simulation based on material property information of the soft and hard tissues. Afterwards, the lunate bone, whose link path is set using the links connected to the first link of the mammary bone, moves and/or moves according to the movement restrictions and characteristics of the second link between the papillary bone and the lunate bone and the first link inside the lunate bone. Determine size and scope.

A joint surgery navigation device can establish links between a plurality of bones included in the hard tissue of a three-dimensional joint model. A link is information that models the movement relationship, such as movement and rotation, between a plurality of bones included in the joint when the joint is moved.

The link will be explained with reference to Figure 7. FIG. 7 shows two link paths established between a plurality of bones included in the hard tissue of the wrist joint as shown in FIG. 4. Figure 7 (a) shows links established between the mammary bone, lunate, and radius, and Figure 7 (b) shows links established between the minor cuneiform bone, navicular bone, and radius.

The joint surgery navigation device may establish a first link within the bones for a plurality of bones and establish a second link between the bones.

The second link may be experimentally established between bones whose movement relationships are determined to be related to each other by measuring the movement, rotation relationship, and change of each bone according to the joint movements of a plurality of subjects.

Referring to Figure 7, it was experimentally determined that the papillary and lunate bones of a plurality of subjects moved in relation to each other based on the radius, and that the minor rhomboid and scaphoid bones moved in relation to each other around the radius. As determined, the joint surgery navigation device can set two link paths between a plurality of bones in the joint 3D model of the wrist joint as shown in FIG. 7. Additional link paths may be set for other bones of the wrist joint.

In the joint surgery navigation device, the first link inside the bone is set to connect the center of gravity of the same bone and the feature point on the surface, and the second link between bones is set to connect the feature point of a plurality of bones, or one of the plurality of bones is set to connect the feature point of the surface. It can be set so that the center of gravity of one bone is connected to the feature point of another bone.

The feature point is a plurality of candidate groups set on the surface of the bone as shown in FIG. 8 according to the structure, shape, and physical relationship between bones, and then a plurality of links connecting the feature points and/or centers of gravity of the plurality of candidate groups with links. By comparing the results of the inverse kinematics motion for the path sets and the results of the actual joint motion, the link path with the least error can be determined as the final links.

In one embodiment, the joint surgery navigation device 100 may set the reference hard tissue not to exercise. For example, as shown in Figure 6, the three-dimensional joint model of the wrist joint is set to be divided into soft tissue and hard tissue, and the hard tissue corresponding to domain 3 (for example, the radius in Figure 7) among the plurality of hard tissues is set as the reference. It can be set so that no change in hard tissue position occurs due to deformation of soft tissue.

In one embodiment, the joint surgery navigation device 100 may unify the coordinates of the feature point and/or the center of gravity, which are the end portions of each link in the link path, and determine the movement direction and movement range of the link. Alternatively, you can receive a joint 3D model with links that already have these characteristics defined.

A method for determining the direction of movement and range of movement of a link will be described with reference to FIGS. 9 to 12.

The joint surgery navigation device 100 may set a link path in which the connection relationship between links is established, as shown in FIG. 9 (a). The joint surgery navigation device 100 may set the first bone and the second bone as movable hard tissues, and the feature points (a0, b0, c0) may be set as reference feature points of the bone so that no change in the position of the hard tissue occurs.

The joint surgery navigation device 100 resets the coordinates of the remaining feature points and/or center of gravity of the link path based on the coordinate system of the feature point of the reference bone (with the feature point of the reference bone as the origin) as shown in FIG. 9 (b). can do. Below, the reference in FIG. 9 will be described as p0, p1, p2, p3, and p4, in order, from the feature point of the bone to the feature point and/or center of gravity along the link path.

Referring to FIG. 10, the joint surgery navigation device 100 may set the coordinate system of the link at p1, p2, p3, and p4, respectively, and determine the X-axis of each link coordinate system as the direction of the link. For example, X1, the X axis of p1, can be set so that the link direction between p0 and p1 follows an extension line. Subsequently, the X-axis of p2, p3, and p4 can be determined in the same way.

The joint surgery navigation device 100 may determine the Z axes of p1, p2, p3, and p4 based on the cross product of the X axes of the link. For example, Z1, the Z axis of p1, can be determined as the external product of X1, the X axis of p1, and X2, the X axis of p2 connected to the link of p1. The Y-axis is determined as the axis where the X-axis and Z-axis are simultaneously orthogonal.

Therefore, ultimately, the plurality of bones included in the joint perform complex movements, rather than simple movements, in relation to each other and in separate directions. To reflect this, the center of gravity and characteristic points of at least two bones among the plurality of bones in the hard tissue are adjusted. The directions of the coordinate axes forming the coordinate system may be set differently. In the case of a wrist joint, among each bone except the reference, the bone at the end of the link located on the traction side moves in parallel in the direction of traction, and the remaining bones perform rotational movement around the Z axis of their own coordinate system. Since the direction of the Z axis is different, the direction of rotation is also different and can reflect complex joint movements.

The joint surgery navigation device 100 may store in advance different link connections and setting relationships for each joint part.

For example, when the joint surgery navigation device 100 receives input that the joint part of the provided 3D image of the patient is a wrist joint, the storage unit 150 creates a link path as shown in FIGS. 7 (a) and 7 (b). By loading and calculating the center of gravity and feature points of each bone from data in which the bones included in each tissue and hard tissue are classified and labeled, a link path as shown in Figures 7 (a) and (b) can be set.

The joint surgery navigation device 100 can calculate the angle (θ) between each It can be determined based on the dot product of the axis. The alpha sign can be given experimentally after observing the movement of bones included in hard tissue depending on the joint.

The joint surgery navigation device 100 calculates the degree of movement due to traction of the distal hard tissue of the hard tissue link path according to the deformation of the soft tissue, and determines the link characteristics (θ, α, link length r, etc.) determined by FIGS. 9 and 10. ) can be reflected in the inverse kinematics to determine the movement direction and range of movement of the link and/or the position of the hard tissues connected to the link included in the same link path (S130).

For example, referring to FIG. 11, p4, the center of gravity of the hard tissue at the end of the hard tissue link path, is p4', which is the target point, according to the movement amounts (dx, dy, dz) by finite element analysis simulation along the traction direction. If it is determined that translation is to be performed, the links of the link paths connected to p4 can be calculated and moved using inverse kinematics. At this time, the direction of movement of each link can be set by performing a rotational movement around the Z-axis of one end of the link in the link paths connected to p4. In particular, in the case shown in Figure 4 where multiple bones are connected in a complex manner, when pulled upward in the direction of the finger based on Figure 4, the link path as shown in Figure 7 (a) based on the radius moves the papillary bone in parallel upward. , the first link inside the lunate bone (link between the lunate bone feature point and the center of gravity of the lunate bone), and the second link between the lunate bone feature point and the papillary bone feature point can each be set to rotate around the Z axis.

The joint surgery navigation device 100 may determine the degree of rotation of the link based on the inverse kinematics and characteristics of each link. Referring to FIG. 12, the coordinate transformation relationship (T) between p0, p1, p2, p3, and p4 can be calculated based on the link characteristics (θ, α, r) and <Equation 1>.

In <Equation 1>, R means rotation and d means movement.

The joint surgery navigation device 100 can determine the degree of rotation of the angle (θ) of each link based on the coordinate transformation relationships (T) and <Equation 2>, until p4 reaches (converges) the target point. Repeat this.

Figure 14 shows the results of calculating the moved positions of each bone in the joint 3D model based on link settings according to an embodiment of the present specification and comparing them with the moved positions of the actual bones. Comparing the results with the results in Figure 1, it can be seen that the movement positions of the bones were calculated to be almost identical to the actual positions.

The joint surgery navigation device 100 can visually display at least a portion of the joint 3D model by overlapping or displaying it separately from the arthroscope 300 image according to the final changed positions of each bone of the hard tissue determined based on the link. (S140).

Referring to FIG. 13, the joint surgery navigation device 100 determines the viewing angle according to the position and posture of the arthroscope 300 camera, and generates a corresponding image from a portion of the joint 3D model corresponding to the viewing angle to determine the joint It can be displayed by overlapping it with the image of the camera of view 300. The joint surgery navigation device 100 recognizes (classifies) each bone in the image of the camera of the arthroscope 300 based on the position of the arthroscope 300 and a three-dimensional joint model in which each tissue and each bone are separated, At least a portion of the 3D model of the joint may be displayed overlapping with the camera image of the arthroscope 300 and the names (labels) of each bone may be displayed together. Therefore, the surgical team can accurately determine where the bones in the image are despite the narrow viewing angle of the arthroscope 300 camera and the narrow surgical space.

In another embodiment, referring to FIG. 16, the joint surgery navigation device 100 displays joint 3 corresponding to the viewing angle of the arthroscope 300 camera image to facilitate determining where the arthroscope 300 camera image is located. An image of the area including part of the dimensional model can be generated and displayed by overlapping the image of the camera of the arthroscope 300 on that part. Likewise, the joint surgery navigation device 100 can recognize each hard tissue and soft tissue in the image of the camera of the arthroscope 300 and the planar image of the generated three-dimensional joint model, and display the names and boundaries by overlapping them.

The present disclosure described above can be implemented as computer-readable code on a program-recorded medium. Computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is. Additionally, the computer may include a processor for each device.

Meanwhile, the program may be specially designed and configured for the present disclosure, or may be known and usable by those skilled in the art of computer software. Examples of programs may include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification (particularly in the claims) of the present disclosure, the use of the term “above” and similar referential terms may refer to both the singular and the plural. In addition, when a range is described in the present disclosure, the invention includes the application of individual values within the range (unless there is a statement to the contrary), and each individual value constituting the range is described in the detailed description of the invention. It's the same.

Unless there is an explicit order or description to the contrary regarding the steps constituting the method according to the present disclosure, the steps may be performed in any suitable order. The present disclosure is not necessarily limited by the order of description of the steps above. The use of any examples or illustrative terms (e.g., etc.) in the present disclosure is merely to describe the present disclosure in detail, and unless limited by the claims, the scope of the present disclosure is limited by the examples or illustrative terms. It doesn't work. In addition, those skilled in the art will recognize that various modifications, combinations and changes may be made according to design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present disclosure should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the claims are within the scope of the spirit of the present disclosure. It will be said to belong to

100: navigation device
200: location tracking device
300: Arthroscopy
310: marker
400: video display device
1000: patient
1100: Joint
1200: Marker

Claims (18)

A method in which at least a portion of each step is performed by a processor, comprising:
Loading a 3D joint model in which soft tissue and hard tissue are divided based on a 3D image of the patient and different physical properties are assigned to the soft tissue and hard tissue;
determining an external force applied to the patient and calculating a deformation of the soft tissue of the three-dimensional joint model based on the external force;
changing the position of the hard tissue in the joint 3D model based on the deformation of the soft tissue in the joint 3D model; and
Comprising the step of visually displaying at least a portion of the joint three-dimensional model based on the position of the arthroscope,
Joint surgery navigation method.
According to claim 1,
Recognizing a marker attached to the patient's skin with an optical camera; and
Further comprising determining the external force based on the recognized movement degree of the marker,
Joint surgery navigation method.
According to clause 2,
The step of visually displaying at least a portion of the joint 3D model includes:
Matching the coordinate system of the three-dimensional joint model with the coordinate system of the arthroscope; and
Based on the position of the arthroscope determined based on the image of the optical camera, at least a portion of the hard tissue of the three-dimensional model of the joint is displayed on a display device, projected by beam projection, or displayed on an augmented reality device worn by the operator. Including the steps of:
Joint surgery navigation method.
According to claim 1,
The hard tissue includes a plurality of bones constituting a joint, and the joint 3D model has a plurality of links set based on movement relationships within and between the plurality of bones,
Joint surgery navigation method.
According to clause 4,
The link inside the plurality of bones connects the center of gravity and a feature point on the surface of the bone,
The link between the plurality of bones connects the characteristic points of the plurality of bones, or connects the center of gravity of one of the plurality of bones and the characteristic point of another bone,
Joint surgery navigation method.
According to clause 4,
The step of changing the position of the hard tissue in the three-dimensional model of the joint is,
changing the position of the center of gravity of a first bone among the plurality of bones based on deformation of the soft tissue;
Translating a first link at which the center of gravity of the first bone is distal, based on a change in position of the center of gravity of the first bone; and
Including rotating other links directly or indirectly connected to the first link with respect to one end of each link,
Joint surgery navigation method.
According to clause 6,
The end of the link furthest from the first link among the plurality of links located in the opposite direction of the first link is fixed in position and does not change.
Joint surgery navigation method.
According to clause 7,
Among the plurality of bones of the hard tissue, the centers of gravity and feature points of at least two bones have different directions of the coordinate axes forming the coordinate system,
Joint surgery navigation method.
According to claim 1,
Receiving a 3D image of the patient;
Separating the soft tissue and the hard tissue from the 3D image of the patient, and dividing the hard tissue into a plurality of bones;
imparting different physical properties including Young's modulus and stiffness to the soft tissue and the hard tissue; and
Comprising the step of generating the joint 3D model by applying the setting relationship of the link to the hard tissue based on the type of joint area from which the 3D image of the patient was obtained,
Joint surgery navigation method.
A storage unit that stores a three-dimensional joint model in which soft tissue and hard tissue are divided based on a three-dimensional image of the patient and different physical properties are given to the soft tissue and hard tissue;
processor; and
A memory electrically connected to the processor and storing at least one code executed by the processor,
When the memory is executed through the processor, the processor
Determine the external force applied to the patient, calculate the deformation of the soft tissue of the joint 3D model based on the external force, and calculate the deformation of the soft tissue of the joint 3D model based on the deformation of the soft tissue of the joint 3D model Changing the position of and storing a code that causes at least a portion of the three-dimensional model of the joint to be visually displayed together with the image taken by the arthroscope based on the position of the arthroscope that takes images of the patient's joint area. doing,
Joint surgery navigation device.
According to claim 10,
The memory allows the processor to:
Further storing a code that causes the marker attached to the patient's skin to be recognized in the image of the optical camera and the external force to be determined based on the recognized movement degree of the marker,
Joint surgery navigation device.
According to claim 11,
The memory allows the processor to:
Matching the coordinate system of the joint 3D model with the coordinate system of the arthroscope,
Based on the position of the arthroscope determined based on the image of the optical camera, at least a portion of the hard tissue of the three-dimensional model of the joint is displayed on a display device, projected by beam projection, or displayed on an augmented reality device worn by the operator. which saves more code, causing
Joint surgery navigation device.
According to claim 10,
The hard tissue includes a plurality of bones constituting a joint, and the joint 3D model has a plurality of links set based on movement relationships within and between the plurality of bones,
Joint surgery navigation device.
According to claim 13,
The link inside the plurality of bones connects the center of gravity and a feature point on the surface of the bone,
The link between the plurality of bones connects the characteristic points of the plurality of bones, or connects the center of gravity of one of the plurality of bones and the characteristic point of another bone,
Joint surgery navigation device.
According to claim 13,
The memory allows the processor to:
The position of the center of gravity of a first bone among the plurality of bones is changed based on the deformation of the soft tissue, and based on the change in the position of the center of gravity of the first bone, the center of gravity of the first bone is the distal end. Further storing code that translates the link and causes other links directly or indirectly connected to the first link to rotate relative to one end of each link,
Joint surgery navigation device.
According to claim 15,
The end of the link furthest from the first link among the plurality of links located in the opposite direction of the first link is fixed in position and does not change.
Joint surgery navigation device.
According to claim 16,
Among the plurality of bones of the hard tissue, the centers of gravity and feature points of at least two bones have different directions of the coordinate axes forming the coordinate system,
Joint surgery navigation device.
According to claim 10,
The memory allows the processor to:
In the provided 3D image of the patient, the soft tissue and the hard tissue are respectively divided, the hard tissue is divided into a plurality of bones, different physical properties including Young's modulus and stiffness are given to the soft tissue and the hard tissue, and the patient Further storing a code that causes the joint 3D model to be generated by applying the setting relationship of the link to the hard tissue based on the type of joint part from which the 3D image was acquired,
Joint surgery navigation device.

Images