JP7012302B2 - Surgery support terminals and programs - Google Patents

Description

The present invention relates to surgical support terminals and programs that support surgical operations, particularly orthopedic surgery in which implants for orthopedic surgery are attached to the body of a patient.

Conventionally, replacement surgery is known in which a knee joint or a hip joint is replaced with an artificial joint made of metal or ceramics. In addition, surgical operations for attaching and fixing a spinal cage, an artificial vertebral body, or the like for fixing, correcting, and stabilizing the spine are also known.
When mounting these implants in place within the body, it is important to place the implants in the optimal shape and dimensions for the patient at the optimal angle and position.

Therefore, in recent years, a preoperative plan that simulates the shape, dimensions, and mounting angle of the implant to be mounted, and if necessary, the amount of osteotomy and the position of osteotomy, etc., using a medical tomographic image of the implant mounting site of the patient. (See, for example, Patent Document 1).
In addition, a navigation system has been developed that optically tracks the position of a marker fixed to the patient's bone during surgery and uses the absolute coordinates of the marker to indicate the amount of osteotomy and the position of osteotomy. ..

Japanese Unexamined Patent Publication No. 2009-82444

However, in the navigation system, the operation time becomes longer because the marker must be attached and detached, and the marker pin is inserted into a site different from the surgical site of the patient's bone, which is invading. There is a problem that the part increases.

Further, the navigation system has a problem that it is expensive, the device itself is large, and the penetration rate is not high. For example, a simple navigation system that can be introduced at a relatively low cost by sacrificing the accuracy of one axial direction (for example, the Z-axis direction) of a three-dimensional space has also been developed. , It is difficult to realize highly accurate surgical support during surgery. For example, in the case of surgery to replace the knee joint with an artificial knee joint, the rotation direction of the tibia cannot be properly specified, and as a result, the mounting accuracy of the jig that guides the osteotomy and implant mounting is reduced, and as a result. , Implant mounting accuracy is reduced.

In addition, the above system provides intraoperative surgical support based on data planned using preoperatively imaged medical tomographic images of the patient's implant attachment site, for example. In the surgery in which the osteotomy is performed, there is also a problem that the surgical support cannot be performed by reflecting the condition of the bone after the osteotomy during the operation.

The present invention has been made in view of such a problem, and an object of the present invention is to provide intraoperative support for surgery with high accuracy and at low cost.

In order to solve the above problems, the surgical support terminal according to one aspect of the present invention is
It is a surgical support terminal equipped with an imaging unit, a display unit, and a posture sensor, and is used when attaching an implant to the femur or tibia.
To guide and support the attachment of the implant to the bone on the reference image relating to the three-dimensional shape of the femur or the tibia to which the patient's implant is to be attached and the three-dimensional model based on the reference image. The first acquisition means for acquiring the jig information including at least one of the orientation and the position of the jig of
An image capture image acquisition means for sequentially acquiring an image captured on the surface of the bone of a patient imaged by the image pickup unit.
When the distal end side of the femur is rotated with the surgical support terminal fixed at a predetermined position of the femur, the center of the femur head is based on the measurement information obtained from the posture sensor. A means for calculating the center of the femoral head, which calculates the absolute coordinates of the position of
The position of the center of the femoral head in the coordinate system of the captured image is specified from the absolute coordinates of the position of the center of the head calculated by the means for calculating the center of the head, and the captured image sequentially acquired by the captured image acquisition means. And the estimation means for estimating the posture of the bone in the captured image by collating with the reference image acquired by the first acquisition means.
Based on the posture of the bone in the captured image estimated by the estimation means and the jig information acquired by the first acquisition means, the captured image sequentially acquired by the captured image acquisition means is obtained. A first display control means for superimposing a first augmented reality image that virtually represents at least one of the orientation and position of the jig and displaying it on the display unit.
It is characterized by having.

Further, the surgical support terminal according to one aspect of the present invention is
It is a surgical support terminal equipped with an imaging unit, a display unit, and a posture sensor, and is used when performing bone processing on the femur or tibia.
A reference image relating to the three-dimensional shape of the femur or the tibia to be processed in the patient's bone, and the direction and position of one bone processing with respect to the bone on the three-dimensional model based on the reference image. A second acquisition means for acquiring bone processing information including at least one of them,
An image capture image acquisition means for sequentially acquiring an image captured on the surface of the bone of a patient imaged by the image pickup unit.
When the distal end side of the femur is rotated with the surgical support terminal fixed at a predetermined position of the femur, the center of the femur head is based on the measurement information obtained from the posture sensor. A means for calculating the center of the femoral head, which calculates the absolute coordinates of the position of
The position of the center of the femoral head in the coordinate system of the captured image is specified from the absolute coordinates of the position of the center of the head calculated by the means for calculating the center of the head, and the captured image sequentially acquired by the captured image acquisition means. And the estimation means for estimating the posture of the bone in the captured image by collating with the reference image acquired by the second acquisition means.
Based on the posture of the bone in the captured image estimated by the estimation means and the bone processing information acquired by the second acquisition means, the captured image sequentially acquired by the captured image acquisition means is obtained. A second display control means for superimposing a second augmented reality image that virtually represents at least one of the directions and positions of the one bone processing and displaying it on the display unit.
It is characterized by having.

The program of one aspect of the present invention is
A computer for a surgical support terminal equipped with an imaging unit, a display unit, and a posture sensor, which is used when attaching an implant to the femur or tibia .
To guide and support the attachment of the implant to the bone on the reference image relating to the three-dimensional shape of the femur or the tibia to which the patient's implant is to be attached and the three-dimensional model based on the reference image. First acquisition means for acquiring jig information including at least one of the orientation and position of the jig,
An image capture image acquisition means for sequentially acquiring an image captured on the surface of the bone of a patient imaged by the image pickup unit.
When the distal end side of the femur is rotated with the surgical support terminal fixed at a predetermined position of the femur, the center of the femur head is based on the measurement information obtained from the posture sensor. A means for calculating the center of the femoral head, which calculates the absolute coordinates of the position of
The position of the center of the femoral head in the coordinate system of the captured image is specified from the absolute coordinates of the position of the center of the head calculated by the means for calculating the center of the head, and the captured image sequentially acquired by the captured image acquisition means. And the estimation means for estimating the posture of the bone in the captured image by collating with the reference image acquired by the first acquisition means.
Based on the posture of the bone in the captured image estimated by the estimation means and the jig information acquired by the first acquisition means, the captured image sequentially acquired by the captured image acquisition means is obtained. A first display control means for superimposing a first augmented reality image that virtually represents at least one of the orientation and position of the jig and displaying it on the display unit.
It is characterized by functioning as.

The program of one aspect of the present invention is
A computer of a surgical support terminal equipped with an imaging unit, a display unit, and a posture sensor, which is used when processing the bones of the femur or tibia .
A reference image relating to the three-dimensional shape of the femur and the tibia to be processed in the patient's bone, and the direction and position of one bone processing with respect to the bone on the three-dimensional model based on the reference image. A second acquisition means for acquiring bone processing information including at least one of them,
An image capture image acquisition means for sequentially acquiring an image captured on the surface of the bone of a patient imaged by the image pickup unit.
When the distal end side of the femur is rotated with the surgical support terminal fixed at a predetermined position of the femur, the center of the femur head is based on the measurement information obtained from the posture sensor. A means for calculating the center of the femoral head, which calculates the absolute coordinates of the position of
The position of the center of the femoral head in the coordinate system of the captured image is specified from the absolute coordinates of the position of the center of the head calculated by the means for calculating the center of the head, and the captured image sequentially acquired by the captured image acquisition means. And the estimation means for estimating the posture of the bone in the captured image by collating with the reference image acquired by the second acquisition means.
Based on the posture of the bone in the captured image estimated by the estimation means and the bone processing information acquired by the second acquisition means, the captured image sequentially acquired by the captured image acquisition means is obtained. A second display control means for superimposing a second augmented reality image that virtually represents at least one of the directions and positions of the one bone processing and displaying it on the display unit.
It is characterized by functioning as.

According to the present invention, intraoperative support for surgery can be performed with high accuracy and at low cost.

It is a figure which shows schematic structure of the operation support system of Embodiment 1 to which this invention was applied. It is a perspective view for demonstrating the total knee arthroplasty in which the surgical support system of FIG. 1 is used. FIG. 3 is a block diagram showing a functional configuration of a hospital terminal constituting the surgery support system of FIG. 1. It is a block diagram which shows the functional configuration of the 1st operation support terminal which constitutes the operation support system of FIG. It is a flowchart which shows an example of the operation which concerns on the 1st operation support processing by the 1st operation support terminal of FIG. It is a figure which shows an example of the display screen of the 1st operation support terminal in the 1st operation support processing of FIG. 5 schematically. It is a perspective view for demonstrating the total hip arthroplasty in which the surgical support system of Embodiment 2 to which this invention is applied is used. It is a block diagram which shows the functional configuration of the 2nd surgery support terminal which constitutes the surgery support system of FIG. 7. It is a flowchart which shows an example of the operation which concerns on the 2nd operation support processing by the 2nd operation support terminal of FIG. It is a figure which shows an example of the display screen of the 2nd operation support terminal in the 2nd operation support process of FIG. 9 schematically.

Hereinafter, a specific embodiment of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

[Embodiment 1]
Hereinafter, the surgical support system 100 of the first embodiment to which the present invention is applied will be described with reference to FIGS. 1 to 6.
FIG. 1 is a diagram schematically showing a schematic configuration of the surgery support system 100 of the first embodiment. Further, FIG. 2 is a perspective view for explaining total knee arthroplasty in which the surgical support system 100 is used.
Although FIG. 2 and FIG. 7 described later show a state in which the patient P on the operating table T is not fixed, it may be fixed so that the doctor can easily perform the surgical operation.

The surgery support system 100 of the first embodiment is a system used for total knee arthroplasty (see FIG. 2), and specifically, as shown in FIG. 1, a hospital terminal 1 installed in a hospital and a hospital terminal 1 It is equipped with a first surgery support terminal 2 that provides intraoperative support for surgery. Further, the hospital terminal 1 and the first surgery support terminal 2 are connected to each other via a wireless communication line (for example, a wireless LAN (Local Area Network) or the like) so that information can be communicated.

<Hospital terminal>
First, the hospital terminal 1 will be described in detail with reference to FIG. FIG. 3 is a block diagram showing a functional configuration of the hospital terminal 1.
The hospital terminal 1 is composed of, for example, a PC (Personal Computer) or the like, and is for performing a preoperative plan for total knee arthroplasty. Specifically, as shown in FIG. 3, the hospital terminal 1 includes a central control unit 101, a memory 102, a storage unit 103, a display unit 104, an operation unit 105, and a communication control unit 106. There is. Further, each part of the hospital terminal 1 is connected via a bus 107.

The central control unit 101 controls each unit of the hospital terminal 1. That is, although not shown, the central control unit 101 includes a CPU (Central Processing Unit), and the CPU reads out a designated program among the system programs and application programs stored in the storage unit 103, and stores the memory. Expand to 102 work areas and execute various processes according to the program. At that time, the CPU of the central control unit 101 stores various processing results in the memory 102, and displays the processing results on the display unit 104 as needed.

The memory 102 is composed of, for example, a DRAM (Dynamic Random Access Memory) or the like, and has a work area for temporarily storing various programs and data read by the CPU of the central control unit 101.

The storage unit 103 is composed of, for example, a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), an HDD (Hard Disk Drive), or the like, and is a storage unit capable of writing and reading data and programs.
Specifically, the storage unit 103 stores the tomographic image data I, the preoperative planning program 103a, and the like.

The tomographic image data I is captured by a medical tomographic image diagnostic device (not shown) such as an X-ray CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging system) device to capture the affected area to be replaced by the artificial joint of the patient P. It is the image data that has been made. That is, the tomographic image data I is, for example, image data in which the tibia B1a and the femur B1b constituting the knee joint of the patient P are imaged by a medical tomographic image diagnostic apparatus before the total knee arthroplasty.
The tomographic image data I may be transmitted from the medical tomographic image diagnostic device to the hospital terminal 1 via a network such as an in-hospital LAN, or may be transmitted to the hospital terminal 1 via a storage medium such as a memory card. It may be the one obtained by.

The preoperative planning program 103a is a program that realizes a function related to preoperative planning processing of artificial joint replacement.
That is, the CPU of the central control unit 101 reads the preoperative planning program 103a from the storage unit 103, and according to the preoperative planning program 103a, the artificial knee joint to be attached (tibial side member, femoral bone side member, etc.) The shape and dimensions of the artificial joint component), the orientation of the tibial intramedullary rod inserted into the tibia B1a and the position of the insertion point, the osteotomy position of the tibia B1a, and the femoral intramedullary rod inserted into the femoral bone B1b. The orientation, the position of the insertion point, the osteotomy position of the tibia B1b, etc. are specified by simulation.

The preoperative planning process is a known technique, and although detailed description thereof will be omitted here, for example, the CPU of the central control unit 101 may use the above-mentioned tomographic image data I to determine the tibia B1a and femur B1b of the patient P. Generates image data representing the three-dimensional shape of. Then, the CPU of the central control unit 101 constructs a coordinate system on the three-dimensional model of the tibia B1a and the femur B1b displayed on the display unit 104, and the tibia side member and the femur side member constituting the artificial knee joint ( Specify by simulating the planned installation position of the artificial joint component).

Further, the CPU of the central control unit 101 uses the tibial side member and the femur with respect to the tibia B1a and the femur B1b on the three-dimensional model of the tibia B1a and the femur B1b displayed on the display unit 104 based on the tomographic image data I. The jig information including at least one of the appropriate orientations and positions of the jig for guiding and supporting the attachment of the bone-side member is simulated and generated. This jig information may be automatically generated by simulation, or the orientation and position of the jig may be finely adjusted based on a predetermined operation of the operation unit 105 by a doctor.

Specifically, the jig information includes, for example, the direction of the intrashineral rod inserted into the tibia B1a to guide and support the attachment of the tibial side member to the tibia B1a, the position of the insertion point, and the inside of the tibia. The orientation and installation position of the tibial osteotomy guide member that guides the osteotomy of the tibial bone B1a with reference to the rod, and the inside of the femoral bone that is inserted into the femoral bone B1b to guide and support the attachment of the femoral bone side member to the femoral bone B1b. The orientation of the rod, the position of the insertion point, the orientation of the femoral osteotomy guide member for guiding the osteotomy of the femoral bone B1b with reference to the rod in the femoral bone, and the installation position are included.

Further, the CPU of the central control unit 101 uses the functional axis of the lower limbs, the position of the center of the head of the femur B1b, the functional axis of the femur B1b, the approximate plane of the anterior cortex of the femur B1b, the functional axis of the tibia B1a, and the varus / valgus. Various intraoperative support parameters such as angle, flexion / extension angle, and rotation angle may be generated.

The display unit 104 is composed of, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or the like. Further, the display unit 104 displays various screens according to the instructions of the display signal output and input from the CPU of the central control unit 101.

The operation unit 105 has, for example, a keyboard, a mouse, or the like composed of data input keys for inputting numerical values, characters, etc., up / down / left / right movement keys for performing data selection, feed operation, etc., and various function keys. I have. Then, the operation unit 105 outputs a predetermined operation signal to the central control unit 101 in response to the operation of the keyboard and the mouse.

The communication control unit 106 performs information communication with the first surgery support terminal 2.
That is, the communication control unit 106 has, for example, a wireless LAN module or the like, and wirelessly communicates directly with the communication control unit 208 of the first surgery support terminal 2 without going through an external access point (fixed base station). Operates in Peer to Peer (ad hoc mode) to build a line. For example, the communication control unit 106 transmits various information such as the communication method, encryption information, channel, and IP address of the wireless communication line to the communication control unit 208 of the first surgery support terminal 2 that is within the wireless communication range. A wireless communication line such as a wireless LAN is constructed with the communication control unit 208 of the first surgery support terminal 2 to transmit and receive signals.
Specifically, the communication control unit 106 uses the tomographic image data I of the patient P and the preoperative planning data D (see FIG. 4) generated by the preoperative planning processing as the communication control unit 208 of the first surgery support terminal 2. Send to.

The communication control unit 106, for example, constructs a wireless communication line with an access point existing within the wireless communication range, and communicates with the communication control unit 208 of the first surgery support terminal 2 via the wireless communication line. It may operate in infrastructure mode in which signals are sent and received between.

<First surgery support terminal>
Next, the first surgery support terminal 2 will be described in detail with reference to FIG. FIG. 4 is a block diagram showing a functional configuration of the first surgery support terminal 2.
The first surgery support terminal 2 is composed of, for example, a smartphone, a tablet, or the like, and is for providing intraoperative support for total knee arthroplasty. Specifically, as shown in FIG. 4, the first surgery support terminal 2 includes a central control unit 201, a memory 202, a storage unit 203, an imaging unit 204, a signal processing unit 205, and a display unit 206. The operation unit 207 and the communication control unit 208 are provided. Further, each part of the first surgery support terminal 2 is connected via a bus 209.

The central control unit 201 controls each unit of the first surgery support terminal 2. That is, although not shown, the central control unit 201 includes a CPU, and the CPU reads out a designated program among the system programs and application programs stored in the storage unit 203 and puts it in the work area of the memory 202. Expand and execute various processes according to the program. At that time, the CPU of the central control unit 201 stores various processing results in the memory 202, and displays the processing results on the display unit 206 as needed.

The memory 202 is composed of, for example, a DRAM or the like, and has a work area for temporarily storing various programs and data read by the CPU of the central control unit 201.

The storage unit 203 is composed of, for example, a flash memory, EEPROM, or the like, and is a storage unit capable of writing and reading data and programs.
Specifically, the storage unit 203 includes the first acquisition program 203a, the captured image acquisition program 203b, the estimation program 203c, the first specific program 203d, the first display control program 203e, the tomographic image data I, the preoperative planning data D, and the like. I remember.

The first acquisition program 203a is a program that realizes the function related to the first acquisition process for acquiring the reference image and the jig information.
That is, the CPU of the central control unit 201 reads the first acquisition program 203a from the storage unit 203, and performs the first acquisition process according to the first acquisition program 203a. Here, the CPU of the central control unit 201 functions as the first acquisition means in cooperation with the first acquisition program 203a.

Specifically, the CPU of the central control unit 201 is transmitted from the hospital terminal 1, received by the communication control unit 208, and stored in the storage unit 203, that is, the tomographic image data I, that is, before the total knee arthroplasty. The tomographic image data I imaged by the medical tomographic image diagnostic apparatus is read out from the storage unit 203 and acquired. Then, the CPU of the central control unit 201 generates and acquires image data of a reference image showing the three-dimensional shapes of the tibia B1a and the femur B1b to which the artificial knee joint of the patient P is to be attached.

Further, the CPU of the central control unit 201 reads the preoperative planning data D stored in the storage unit 203 from the storage unit 203, which is transmitted from the hospital terminal 1 and received by the communication control unit 208. That is, the CPU of the central control unit 201 guides and supports the attachment of the tibia-side member and the femur-side member to the tibia B1a and the femur B1b on the three-dimensional model of the tibia B1a and the femur B1b based on the tomographic image data I. Preoperative planning data D including the jig information including at least one of the orientation and the position of the jig for the tibia is acquired.

The captured image acquisition program 203b is a program that realizes a function related to the captured image acquisition process for sequentially acquiring captured images F (see FIG. 6A and the like).
That is, the CPU of the central control unit 201 reads the captured image acquisition program 203b from the storage unit 203, and performs the captured image acquisition process according to the captured image acquisition program 203b. Here, the CPU of the central control unit 201 functions as a captured image acquisition means in cooperation with the captured image acquisition program 203b.

Specifically, the CPU of the central control unit 201 acquires an image F in which the surface of the bone B of the patient P on the operating table T is imaged. For example, during total knee arthroplasty, when the tibia B1a and femur B1b of patient P on the operating table T are imaged by the imaging unit 204 so that the joint surface of the knee joint can be seen from the medial side, the imaging unit 204 The image pickup control unit reads out a frame image for each screen from the image pickup area of the electronic image pickup unit and outputs the frame image to the signal processing unit 205. At this time, it is preferable that the tibia B1a and the femur B1b are present at a predetermined position (for example, substantially the center) within the angle of view and are imaged by the imaging unit 204 so as to be captured in a predetermined size. Then, the signal processing unit 205 performs various image signal processing on the analog value signal of the frame image to generate a digital value luminance signal Y and a color difference signal Cb, Cr (YUV data) in the memory 202. Output. The CPU of the central control unit 201 sequentially acquires image data (YUV data) of digital values of the frame image (captured image F) from the memory 202.

As will be described later, the captured image F is a part corresponding to a feature point used for collation with at least a reference image of bone B such as tibia B1a and femur B1b (for example, proximal anterior medial part of tibia and femur). It suffices if the distal anterior medial part, etc.) is imaged.

The estimation program 203c is a program that realizes a function related to an estimation process for estimating the posture of the bone B of the patient P in the captured image F.
That is, the CPU of the central control unit 201 reads the estimation program 203c from the storage unit 203, and performs estimation processing according to the estimation program 203c. Here, the CPU of the central control unit 201 functions as an estimation means in cooperation with the estimation program 203c.

Specifically, the CPU of the central control unit 201 obtains the captured images F sequentially acquired by the captured image acquisition process (for example, the captured image F representing the surface shape of the tibia B1a and the femur B1b) by the first acquisition process. It is collated with the acquired reference image (for example, a reference image representing the tibia B1a or the femur B1b), and the posture of the bone B of the patient P in the captured image F is estimated based on the collation result.

Here, since the method of collating the captured image F with the reference image is a known technique, detailed description thereof will be omitted. For example, the CPU of the central control unit 201 is a reference representing the three-dimensional shape of the bone B. Multiple images (3D / 2D converted images) obtained by projecting an image onto an arbitrary two-dimensional plane are generated, and feature quantities of feature points extracted from the captured image F (for example, SIFT (Scale-Invariant Features Transform) feature quantities and SURF). (Speeded Up Robust Features) Search for images with the highest degree of similarity (features, etc.). At this time, the 3D / 2D converted image or the captured image F may be enlarged or reduced as needed.

Further, the CPU of the central control unit 201 is the coordinate position of each point constituting the bone B on the three-dimensional model of the bone B corresponding to the image having the highest similarity, and the patient P in the captured image F. Based on the coordinate positions and the like of each point constituting the bone B, a predetermined coordinate conversion formula is solved to estimate a matrix representing the posture and positional relationship of the bone B of the patient P with respect to the first surgery support terminal 2. At this time, bundle adjustment may be performed to improve the estimation accuracy of the parameters of the geometric model from the image.

As the captured image F, the captured image F captured by the imaging unit 204 so that the bone B of the patient P exists at a predetermined position (for example, substantially the center) within the angle of view and is captured in a predetermined size. It is considered that it is not always necessary to estimate the position of the bone B in the captured image F, but various processes (first specific process, first display control process, etc .; described later) using the estimated result are performed. In order to perform more appropriately, it is preferable to estimate the posture and position of the bone B in the captured image F.

Further, in collation between the captured image F and the reference image, for example, the shape and state of the bone B to be imaged, the orientation and position of the first surgery support terminal 2 at the time of imaging, the brightness of the imaging environment, the direction of light, and the like. Depending on the situation, it may not be possible to properly extract feature points and feature quantities. Therefore, the characteristic shape portion of the bone B (for example, the proximal anterior medial part of the tibia, the distal anteromedial part of the femur, etc.) is designated and registered in the reference image in advance, and the imaging unit 204 is registered. By instructing the characteristic shape portion using a predetermined pointer (not shown) or the like at the time of imaging the bone B by the above, it is possible to more appropriately extract the feature points and feature amounts. May be.

In addition, instead of collating the captured image F using the feature amount of the feature point with the reference image, or in combination with each other, the bone B is previously excised at a predetermined position (for example, by osteotomy) using a laser scalpel or the like. A marker for alignment may be attached to the position to be performed, etc.), the marker may be imaged to estimate the posture, and the posture of the bone B of the patient P may be estimated from the estimation result.

The first specific program 203d is a program that realizes a function related to the first specific process for specifying the direction and position of a jig (not shown).
That is, the CPU of the central control unit 201 reads the first specific program 203d from the storage unit 203, and performs the first specific process according to the first specific program 203d. Here, the CPU of the central control unit 201 functions as the first specific means in cooperation with the first specific program 203d.

Specifically, the CPU of the central control unit 201 is the posture (preferably posture and position) of the bone B (for example, tibia B1a, femur B1b, etc.) of the patient P in the captured image F estimated by the estimation process. ) And the preoperative planning data D including the jig information acquired by the first acquisition process, and at least one of the orientation and the position of the jig with respect to the bone B of the patient P in the captured image F is specified. do.

For example, in the case of the tibia B1a, the CPU of the central control unit 201 determines the posture of the tibia B1a in the captured image F and the tibia inserted into the tibia B1a on the three-dimensional model of the tibia B1a included in the jig information. Based on the orientation of the intramedullary rod and the position of the insertion point, the coordinate system of the tibia B1a in the image F is aligned with the coordinate system of the three-dimensional model of the tibia B1a, and the tibia in the image F is aligned. The orientation of the intratibial rod and the position of the insertion point with respect to the tibia B1a in the coordinate system of B1a are specified. Further, for example, when the intratibial rod is inserted into the tibia B1a, the CPU of the central control unit 201 determines the posture of the tibia B1a in the captured image F and the tertiary of the tibia B1a included in the jig information. Based on the orientation and installation position of the tibial osteotomy guide member on the original model, the coordinate system of the tibia B1a in the captured image F and the coordinate system of the three-dimensional model of the tibia B1a are aligned to perform the captured image. The orientation and installation position of the tibial osteotomy guide member with respect to the tibia B1a in F are specified.

Further, in the case of the femur B1b, the processing content is substantially the same, and although detailed description thereof is omitted here, the orientation of the femoral intramedullary rod and the position of the insertion point, the orientation of the femoral osteotomy guide member, and the installation position For example, the position of the center of the head of the femur B1b in the coordinate system of the femur B1b in the captured image F may be required.
In this case, for example, the first surgical support terminal 2 is fixed at a predetermined position on the distal end side of the femur B1b of the patient P via a predetermined fixing mechanism (for example, screwed into a position to be excised by osteotomy). , The position that is not excised is clamped and fixed; (not shown), and in this state, the angular velocity, angular acceleration, etc. when the distal end side of the femur B1b is rotated are determined by the first surgery support terminal 2. By measuring using an angular velocity sensor (not shown), the absolute coordinates of the position of the center of the head of the femur B1b are calculated. Then, the position of the center of the femur B1 in the coordinate system of the femur B1b in the captured image F may be specified from the absolute coordinates of the position of the center of the femur B1b calculated.

The position of the center of the head of the femur B1b in the coordinate system of the femur B1b in the captured image F by this method is specified when the captured image F and the reference image are collated in the above estimation process. This also makes it possible to more accurately estimate the posture of the femur B1b in the captured image F.

The first display control program 203e is a program that realizes a function related to the first display control process for displaying an augmented reality image on the display panel 206a of the display unit 206 by using augmented reality (AR) technology.
That is, the CPU of the central control unit 201 reads the first display control program 203e from the storage unit 203, and performs the first display control process according to the first display control program 203e. Here, the CPU of the central control unit 201 functions as the first display control means in cooperation with the first display control program 203e.
Specifically, the CPU of the central control unit 201 determines the posture (preferably the posture and position) of the bone B (for example, tibial bone B1a, femoral bone B1b, etc.) in the captured image F estimated by the estimation process. Based on the jig information acquired by the first acquisition process, at least one of the orientation and position of the jig is virtually set to the captured image F sequentially acquired by the captured image acquisition process using augmented reality technology. The first augmented reality image AR1 represented by the above is superimposed and displayed on the display panel 206a (see FIGS. 6 (a) and 6 (b)). More specifically, the CPU of the central control unit 201 virtually represents the orientation and position of the jig with respect to the bone B of the patient P in the captured image F specified by the first specific process, the first augmented reality image AR1. Is superimposed on the captured image F and displayed on the display panel 206a.

For example, in the case of the tibia B1a, the CPU of the central control unit 201 virtually represents the orientation of the intrashinal bone marrow rod with respect to the tibial bone B1a in the captured image F and the position of the insertion point of the intrashinal bone marrow rod. The image AR1 is superimposed on the captured image F and displayed on the display panel 206a (see FIG. 6A). Further, for example, when the intratibial rod is inserted into the tibia B1a, the CPU of the central control unit 201 virtually represents the direction and position of the tibial osteotomy guide member with respect to the tibia B1a in the captured image F. The first augmented reality image (not shown) is superimposed on the captured image F and displayed on the display panel 206a.

Further, in the case of the femur B1b, the processing content is substantially the same, and detailed description thereof will be omitted here. However, for example, as shown in FIG. 6B, the CPU of the central control unit 201 is in the captured image F. The first augmented reality image AR1 that virtually represents the orientation of the femoral intramedullary rod with respect to the femur B1b and the position of the insertion point of the femoral intramedullary rod is superimposed on the captured image F and displayed on the display panel 206a.

As the first augmented reality image AR1, as shown in FIGS. 6 (a) and 6 (b), a long cylindrical shape is exemplified, but the shape is not limited to this as an example. And dimensions can be changed as appropriate. Further, since the first augmented reality image AR1 is superimposed and displayed on the captured image F, it is in a semi-transparent state as shown in FIGS. 6A and 6B, or only the outer shape is identified. It is preferable that the transparent state displayed as possible does not interfere with the visual recognition of the bone B in the captured image F by the user (for example, a doctor or the like).

In addition, in FIG. 6A, the illustration of the portion other than the tibia B1a of the patient P is omitted, and similarly, in FIG. 6B, the illustration of the portion other than the femur B1b of the patient P is shown. Is omitted, but in reality, the tissues around the tibia B1a and the femur (for example, cartilage, tendon, ligament, skin, etc.) are also imaged by the imaging unit 204 and displayed on the display panel 206a. Become.

The imaging unit 204 images the surface of the bone B of the patient P on the operating table T. Specifically, the image pickup unit 204 includes, for example, a lens unit, an electronic image pickup unit, and an image pickup control unit, although not shown.

The lens unit is composed of a plurality of lenses (not shown) such as a zoom lens and a focus lens, for example.
The electronic imaging unit is composed of, for example, an image sensor (not shown) such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal-oxide Semiconductor), and a two-dimensional image signal is obtained from an optical image that has passed through various lenses of the lens unit. Convert to.
The image pickup control unit scans and drives the electronic image pickup unit by a timing generator, a driver (not shown), etc., and converts the optical image formed by the lens unit into a two-dimensional image signal by the electronic image pickup unit at predetermined intervals. Then, the frame image for each screen is read out from the image pickup area of the electronic image pickup unit and output to the signal processing unit 205.

The signal processing unit 205 performs various image signal processing on the analog value signal of the frame image transferred from the electronic imaging unit.
Specifically, the signal processing unit 205 appropriately adjusts the gain of each RGB color component with respect to the analog value signal of the frame image, holds the sample with a sample hold circuit (not shown), and performs A / D conversion. It is converted into digital data by a device (not shown), and color process processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown) to perform digital value brightness signal Y and color difference signals Cb, Cr (YUV). Data) is generated.
Further, the signal processing unit 205 outputs the generated luminance signal Y and the color difference signals Cb and Cr to the memory 202 used as the buffer memory.

The display unit 206 displays various information in the display area under the control of the CPU of the central control unit 201, for example. Specifically, the display unit 206 includes, for example, a VRAM (Video Random Access Memory), a VRAM controller, a digital video encoder, and the like, and displays an application screen based on the execution of various application programs by the CPU of the central control unit 201. Generate and output the generated application screen data to the display panel 206a.
The display panel 206a is composed of, for example, a liquid crystal display panel, an organic EL display panel, or the like.

The operation unit 207 is for performing a predetermined operation of the first surgery support terminal 2. Specifically, the operation unit 207 includes a power button related to power ON / OFF operation, a shutter button related to an imaging instruction, a button related to selection instructions such as various modes and functions (all not shown). There is. Then, when various buttons are operated by the user, the operation unit 207 outputs an operation instruction corresponding to the operated button to the central control unit 201. The central control unit 201 causes each unit to perform a predetermined operation (for example, imaging a subject) according to an operation instruction output from the operation unit 207 and input.
The operation unit 207 may have a touch panel (not shown) provided integrally with the display panel 206a of the display unit 206.

The communication control unit 208 performs information communication with the hospital terminal 1.
That is, the communication control unit 208 has, for example, a wireless LAN module or the like substantially like the communication control unit 106 of the hospital terminal 1, and operates in Peer to Peer (ad hoc mode) or infrastructure mode.
Specifically, the communication control unit 208 receives the tomographic image data I and the preoperative planning data D of the patient P transmitted from the communication control unit 106 of the hospital terminal 1. The tomographic image data I and the preoperative planning data D received by the communication control unit 208 are output to the storage unit 203 and stored in the storage unit 203.

<1st intraoperative support processing>
The first intraoperative support process in total knee arthroplasty will be described below with reference to FIGS. 5 and 6.
FIG. 5 is a flowchart showing an example of an operation related to the first intraoperative support process using the first surgery support terminal 2. 6 (a) and 6 (b) are diagrams schematically showing an example of a display screen displayed on the display panel 206a of the first surgery support terminal 2 in the first intraoperative support process.

In the following description, preoperative planning processing for total knee arthroplasty is performed in advance on the hospital terminal 1, and tomographic image data I and preoperative planning data D are converted from the hospital terminal 1 to the first surgery support terminal 2. It is assumed that the data is transmitted to the storage unit 203 and stored in the storage unit 203.

As shown in FIG. 5, first, the CPU of the central control unit 201 includes a reference image showing the three-dimensional shapes of the tibia B1a and the femur B1b to which the artificial knee joint of the patient P is to be attached, and jig information. Preoperative plan data D is acquired (step S1; first acquisition process).
Specifically, the CPU of the central control unit 201 reads and acquires the tomographic image data I from the storage unit 203 according to the first acquisition program 203a, and is a reference representing the three-dimensional shapes of the tibia B1a and the femur B1b of the patient P. Generate and acquire image data of an image. Further, the CPU of the central control unit 201 guides and supports the attachment of the artificial joint component to the tibia B1a and the femur B1b, such as an intratibial rod, a tibial osteotomy guide member, an intrafemoral rod, and a femur osteotomy guide. Preoperative planning data D including jig information related to a jig such as a member is acquired.

Next, the CPU of the central control unit 201 supports one of the tibia B1a and the femur B1b designated based on a predetermined operation of the operation unit 207 by the user (for example, the tibia B1a) during the first operation. It is designated as the target bone to be processed (step S2).

Subsequently, the CPU of the central control unit 201 sequentially acquires the captured image F in which the surface of the target bone of the patient P is captured (step S3; captured image acquisition process).
Specifically, when the imaging unit 204 of the target bone (for example, tibia B1a or the like) of the patient P on the operating table T starts imaging, the signal processing unit 205 captures a frame image of the surface of the target bone. Image data (YUV data) of digital values of (captured image F) is sequentially generated and output to the memory 202. The CPU of the central control unit 201 sequentially acquires the image data of the captured image F sequentially generated by the signal processing unit 205 from the memory 202 according to the captured image acquisition program 203b.

After that, the CPU of the central control unit 201 estimates the posture of the target bone of the patient P in the captured image F (step S4; estimation process).
Specifically, the CPU of the central control unit 201 first acquires the captured image F (for example, the captured image F representing the surface shape of the tibia B1a) sequentially acquired by the captured image acquisition process according to the estimation program 203c. A matrix showing the posture and positional relationship of the target bone of the patient P (for example, tibia B1a, etc.) with respect to the first surgery support terminal 2 is collated with the reference image (for example, a reference image representing tibia B1a, etc.) acquired by presume.

Next, the CPU of the central control unit 201 specifies the direction and position of the jig with respect to the target bone of the patient P in the captured image F (step S5; first specifying process).
Specifically, the CPU of the central control unit 201 determines the posture of the target bone (for example, tibial bone B1a, etc.) in the captured image F estimated by the estimation process according to the first specific program 203d, and the first acquisition process. Based on the preoperative planning data D including the acquired jig information, the coordinate system of the target bone in the captured image F and the coordinate system of the three-dimensional model of the target bone are aligned in the captured image F. The orientation and position of the jig with respect to the target bone in the coordinate system of the target bone (for example, the orientation of the rod in the tibial bone and the position of the insertion point) are specified.

Subsequently, the CPU of the central control unit 201 superimposes the first augmented reality image AR1 that virtually represents the orientation and position of the jig with respect to the target bone on the captured image F and displays it on the display panel 206a. (Step S6; first display control process).
Specifically, the CPU of the central control unit 201 is a jig for the target bone (for example, tibia B1a, etc.) of the patient P in the captured image F specified by the first specific process according to the first display control program 203e. The first augmented reality image AR1 that virtually represents the orientation and position is superimposed on the captured image F and displayed on the display panel 206a.

For example, when the target bone is the tibia B1a, the CPU of the central control unit 201 determines the orientation of the tibial intramedullary rod with respect to the tibial bone B1a in the image F and the direction of the tibial intramedullary rod before the insertion of the tibial intramedullary rod. The first augmented reality image AR1 that virtually represents the position of the insertion point is superimposed on the captured image F and displayed on the display panel 206a (see FIG. 6A), and after the insertion of the intratibial rod. A first augmented reality image (not shown) that virtually represents the orientation and position of the tibial osteotomy guide member with respect to the tibia B1a in the captured image F is superimposed on the captured image F and displayed on the display panel 206a.
This allows the doctor to visually recognize the first augmented reality image AR1 and grasp the orientation of the intratibial rod with respect to the tibia B1a, the position of the insertion point, and the orientation and position of the tibial osteotomy guide member.

After that, the CPU of the central control unit 201 determines whether or not there is another bone B as a target bone among the tibia B1a and the femur B1b (step S7).
Here, if it is determined that there is another bone B to be the target bone (step S7; YES), the CPU of the central control unit 201 is not designated as the previous target bone among the tibia B1a and the femur B1b. After designating the other bone B (for example, femur B1b or the like) as a new target bone (step S8), the process returns to step S3.

Then, the CPU of the central control unit 201 performs each process of steps S3 to S6 in substantially the same manner as described above, and in the first display control process of step S6, the target bone of the patient P in the captured image F (for example, for example. The first augmented reality image AR1 that virtually represents the orientation and position of the jig with respect to the femur B1b, etc.) is superimposed on the captured image F and displayed on the display panel 206a.

For example, when the target bone is the femur B1b, the CPU of the central control unit 201 determines the orientation of the femur intramedullary rod with respect to the femur B1b in the captured image F and the femoral intramedullary bone before the insertion of the femoral intramedullary rod. The first augmented reality image AR1 that virtually represents the position of the insertion point of the rod is superimposed on the captured image F and displayed on the display panel 206a (see FIG. 6B), and the insertion of the rod in the femur is also performed. Later, a first augmented reality image (not shown) that virtually represents the orientation and position of the femur osteotomy guide member with respect to the femur B1b in the captured image F is superimposed on the captured image F and displayed on the display panel 206a. ..
Thereby, the doctor can visually recognize the first augmented reality image AR1 and grasp the direction of the rod in the femur with respect to the femur B1b, the position of the insertion point, and the direction and position of the femoral osteotomy guide member.

On the other hand, if it is determined in step S7 that there is no other bone B to be the target bone (step S7; NO), the CPU of the central control unit 201 ends the first intraoperative support process.

As described above, according to the operation support system 100 of the first embodiment, the implant of the patient P (for example, the artificial knee joint, etc.) is scheduled to be attached to the first operation support terminal 2 during the operation of the artificial knee joint replacement. The image F in which the surface of the bone B is imaged is sequentially acquired, the posture of the bone B in the image F is estimated, and the posture of the bone B in the estimated image F and the bone B are obtained. In the jig information including the direction and position of the jig (for example, intramedullary rod, osteotomy guide member, etc.) for guiding and supporting the attachment of the implant to the bone B on the three-dimensional model based on the reference image of. Based on this, the first augmented reality image AR1 that virtually represents the orientation and position of the jig is superimposed on the sequentially acquired image F and displayed on the display unit 206, so that an expensive navigation system can be used as in the conventional case. Even if it is not used, the doctor can properly grasp the direction and position of the jig with respect to the bone B only by visually recognizing the first augmented reality image AR1 displayed on the display unit 206, and the intraoperative support for surgical operation is high. It can be done with accuracy and at low cost. That is, the doctor sequentially grasps the direction and position of the jig with respect to the bone B in consideration of the direction and position of the bone B (particularly, rotation, etc.) that changes according to the state of the patient P's position on the operating table T. As a result, the artificial knee joint can be attached more accurately.
In particular, as the first surgery support terminal 2, for example, a general terminal such as a smartphone or a tablet can be applied, and the introduction cost of the surgery support system 100 can be further reduced.
In addition, the navigation system eliminates the need for the required markers, which not only allows for shorter surgery times, but also allows total knee arthroplasty to be performed with less invasiveness.

[Embodiment 2]
Hereinafter, the surgical support system 100 of the second embodiment to which the present invention is applied will be described with reference to FIGS. 7 to 10.
The points other than those described below are substantially the same as those in the first embodiment, and detailed description thereof will be omitted.

FIG. 7 is a perspective view for explaining total knee arthroplasty in which the surgical support system 100 of the second embodiment is used.
As shown in FIG. 7, the surgery support system 100 of the second embodiment is a system used for total hip arthroplasty, and is a hospital terminal connected to enable information communication via a wireless communication line as in the first embodiment. 1 and a second surgery support terminal 2A (see FIG. 8) are provided.
Note that FIG. 7 illustrates the patient P in the lateral decubitus position on the operating table T, but the present invention is not limited to this, and the patient P may be in the supine position, for example.

<Hospital terminal>
The hospital terminal 1 is substantially the same as the first embodiment except that it is for performing a preoperative plan for total hip arthroplasty.
That is, for example, the storage unit 103 of the hospital terminal 1 has tomographic image data in which the pelvis (not shown) and the femur B2 constituting the hip joint of the patient P are imaged by a medical tomographic image diagnostic device before the total hip arthroplasty. I and the preoperative planning program 103a for preoperative planning processing of total hip arthroplasty are stored.

The preoperative planning program 103a is a program that realizes a function related to preoperative planning processing of total hip arthroplasty.
That is, the CPU of the central control unit 101 reads the preoperative planning program 103a from the storage unit 103, and according to this preoperative planning program 103a, the artificial hip joint (pelvic side member (pelvic cup) and femur) to be attached is to be attached. The shape and dimensions of the side member (artificial joint component such as the femur stem), the reaming position and reaming amount of the acetabulum, the osteotomy position of the femur B2, and the like are specified by simulation.

Specifically, substantially similar to the first embodiment, the CPU of the central control unit 101 generates image data representing the three-dimensional shape of the pelvis and the femur B2 of the patient P from the tomographic image data I described above. Then, the CPU of the central control unit 101 constructs a coordinate system on the three-dimensional model of the pelvis and the femur displayed on the display unit 104, and the pelvis side member and the femur side member (artificial joint) constituting the artificial hip joint. Specify by simulating the planned installation position of the components).

Further, the CPU of the central control unit 101 performs one bone processing (bone cutting) for the pelvis and the femur B2 on the three-dimensional model of the pelvis and the femur B2 displayed on the display unit 104 based on the tomographic image data I. And cutting) to simulate and generate bone processing information including at least one of the appropriate orientations and positions. Further, the bone processing information is displayed on the display unit 104, and the femur with respect to the femur B2 after one bone processing on the three-dimensional model of the femur B2 in a state where one bone processing is virtually performed. Includes at least one of the orientations and positions of other bone processes for attachment of the bone side member. This bone processing information may be automatically generated by simulation, or even if the direction and position of one bone processing or another bone processing is finely adjusted based on a predetermined operation of the operation unit 105 by a doctor. good.

Specifically, one bone processing includes, for example, reaming of the acetabulum of the pelvis, osteotomy of the head of the femur B2, and the like, and examples of other bone processing include reaming of the femur B2. Be done.
In addition, the CPU of the central control unit 101 generates various intraoperative support parameters such as the inclination of the anterior pelvic plane of the pelvis, the size and orientation of the acetabulum, the position of the center of the femur B2, and the anatomical axis of the femur B2. May be.

<Second surgery support terminal>
Next, the second surgery support terminal 2A will be described in detail with reference to FIG. FIG. 8 is a block diagram showing a functional configuration of the second surgery support terminal 2A.
The second surgical support terminal 2A is substantially the same as the first embodiment except that it is for performing intraoperative support for total hip arthroplasty. Specifically, as shown in FIG. 8, the storage unit 203 of the second surgery support terminal 2A has a second acquisition program 203f, an image capture image acquisition program 203b, an estimation program 203c, a second specific program 203g, and a second display control. The program 203h, tomographic image data I, preoperative planning data D, etc. are stored.

The second acquisition program 203f is a program that realizes the function related to the second acquisition process for acquiring the reference image and the bone processing information.
That is, the CPU of the central control unit 201 reads the second acquisition program 203f from the storage unit 203, and performs the second acquisition process according to the second acquisition program 203f. Here, the CPU of the central control unit 201 functions as a second acquisition means in cooperation with the second acquisition program 203f.

Specifically, the CPU of the central control unit 201 is transmitted from the hospital terminal 1, received by the communication control unit 208, and stored in the storage unit 203, substantially similar to the first acquisition process of the first embodiment. The tomographic image data I, that is, the tomographic image data I imaged by the medical tomographic image diagnostic apparatus before the artificial hip joint replacement operation is read out from the storage unit 203 and acquired. Then, the CPU of the central control unit 201 generates and acquires image data of a reference image showing the three-dimensional shape of the pelvis and the femur B2 to which the artificial hip joint of the patient P is to be attached. This reference image includes a reference image relating to a three-dimensional shape in which one bone is virtually processed on the bone B (for example, the femur B2) based on the bone processing information described later.

Further, the CPU of the central control unit 201 reads the preoperative planning data D stored in the storage unit 203 from the storage unit 203, which is transmitted from the hospital terminal 1 and received by the communication control unit 208. That is, the CPU of the central control unit 201 uses at least one of the orientations and positions of one bone processing with respect to the pelvis and the femur B2 on the three-dimensional model of the pelvis and the femur B2 based on the tomographic image data I, and , A technique that includes bone processing information including at least one of the orientations and positions of other bone processing with respect to the femur B2 on a three-dimensional model of the femur B2 in a virtually one bone processing state. Acquire the previous plan data D.

The captured image acquisition program 203b is a program that realizes a function related to the captured image acquisition process for sequentially acquiring captured images F (see FIG. 10A and the like), substantially similar to the first embodiment.
That is, for example, when the pelvis or femur B2 of the patient P on the operating table T is imaged by the imaging unit 204 during the artificial hip joint replacement operation, the imaging control unit of the imaging unit 204 is 1 from the imaging region of the electronic imaging unit. The frame image is read out for each screen and output to the signal processing unit 205. The signal processing unit 205 performs various image signal processing on the analog value signal of the frame image, generates a digital value luminance signal Y and a color difference signal Cb, Cr (YUV data), and outputs the digital value to the memory 202. .. The CPU of the central control unit 201 sequentially acquires image data (YUV data) of digital values of the frame image (captured image F) from the memory 202.

Further, after one bone processing is performed on the bone B of the patient P, the CPU of the central control unit 201 uses the captured image acquisition process to capture the captured image of the bone B after the one bone processing. The image data of F is sequentially acquired.

The estimation program 203c is a program that realizes a function related to an estimation process for estimating the posture of the bone B of the patient P in the captured image F, substantially similar to the first embodiment.
That is, specifically, the CPU of the central control unit 201 acquires the captured image F (for example, the captured image F representing the surface shape of the pelvis and the femur B2) sequentially acquired by the captured image acquisition process for the second acquisition process. The posture of the bone B of the patient P in the captured image F is estimated based on the collation result with the reference image (for example, the reference image representing the pelvis or the femur B2) acquired by.

Further, after one bone processing was performed on the bone B of the patient P, the CPU of the central control unit 201 acquired the captured image F sequentially acquired by the captured image acquisition process and the second acquisition process. By collating with the reference image, the posture of the bone B after one bone processing in the captured image F is estimated. Here, the process of estimating the posture of the bone B after one bone processing is substantially the same as the process of estimating the posture of the bone B before one bone processing described above, and detailed description thereof will be omitted here. For example, the CPU of the central control unit 201 produces a reference image showing a three-dimensional shape in a state in which one bone is virtually processed on the bone B (for example, the femur B2) based on the bone processing information. A plurality of images projected on an arbitrary two-dimensional plane are generated and collated with the captured image F.

The second specific program 203g is a program that realizes the function related to the second specific processing that specifies the direction and position of the bone processing.
That is, the CPU of the central control unit 201 reads the second specific program 203g from the storage unit 203, and performs the second specific processing according to the second specific program 203g. Here, the CPU of the central control unit 201 functions as a second specific means in cooperation with the second specific program 203 g.

Specifically, the CPU of the central control unit 201 is the posture (preferably the posture and position) of the bone B (for example, the pelvis, the femur B2, etc.) of the patient P in the captured image F estimated by the estimation process. And at least one of the directions and positions of one bone processing with respect to the bone B of the patient P in the captured image F based on the preoperative planning data D including the bone processing information acquired by the second acquisition process. Identify. For example, in the case of the femoral bone B2, the CPU of the central control unit 201 refers to the posture of the femoral bone B2 in the captured image F and the femoral bone B2 on the three-dimensional model of the femoral bone B2 included in the bone processing information. Based on the orientation and position of the osteotomy (one bone processing), the coordinate system of the femoral bone B2 in the captured image F and the coordinate system of the three-dimensional model of the femoral bone B2 are aligned, and the captured image F is performed. The orientation and position of the osteotomy of the head of the femoral bone B2 in the coordinate system of the femoral bone B2 inside is specified.

Further, the CPU of the central control unit 201 has acquired the posture of one bone B (for example, femur B2, etc.) after bone processing in the captured image F estimated by the estimation process and the second acquisition process. Based on the preoperative planning data D including the bone processing information, at least one of the orientations and positions of the other bone processing with respect to one post-bone processing bone B in the captured image F is specified. For example, when the osteotomy of the head of the femoral bone B2 is performed as one bone processing, the CPU of the central control unit 201 uses the posture of the femoral bone B2 after the osteotomy and the bone processing information in the captured image F. Reaming (other bone processing) for attachment of the femoral side member to the femoral B2 after one bone processing on the three-dimensional model of the femoral bone B2 in the state of being virtually one bone processed included. ), The coordinate system of the femoral bone B2 after osteotomy in the captured image F and the coordinate system of the three-dimensional model of the femoral bone B2 are aligned based on the orientation and position of the bone in the captured image F. The orientation and position of the reaming of the femoral bone B2 in the coordinate system of the femoral bone B2 after cutting are specified.

Further, in the case of the pelvis, the processing content is substantially the same, and although detailed description thereof is omitted here, the CPU of the central control unit 201 uses the posture of the pelvis in the captured image F and the pelvis included in the bone processing information. The position of the pelvic coordinate system in the captured image F and the position of the coordinate system of the pelvic three-dimensional model based on the orientation and position of the reaming (one bone processing) of the pelvis of the pelvis on the three-dimensional model of the above. The alignment is performed to specify the direction and position of the reaming of the pelvic acetabulum in the coordinate system of the pelvis in the captured image F.

The second display control program 203h is a program that realizes a function related to the second display control process for displaying an augmented reality image on the display panel 206a of the display unit 206 by using the augmented reality technology.
That is, the CPU of the central control unit 201 reads the second display control program 203h from the storage unit 203, and performs the second display control process according to the second display control program 203h. Here, the CPU of the central control unit 201 functions as a second display control means in cooperation with the second display control program 203h.

Specifically, the CPU of the central control unit 201 has the posture (preferably the posture and position) of the bone B (for example, the pelvis, the femur B2, etc.) in the captured image F estimated by the estimation process, and the first. 2 Based on the bone processing information acquired by the acquisition process, at least one of the orientation and position of one bone processing is virtualized on the captured image F sequentially acquired by the captured image acquisition process using the augmented reality technology. The second augmented reality image AR2 to be represented is superimposed and displayed on the display panel 206a. More specifically, the CPU of the central control unit 201 uses the second augmented reality image AR2 that virtually represents the direction and position of one bone processing with respect to the bone B in the captured image F specified by the second specific processing. It is superimposed on the captured image F and displayed on the display panel 206a. For example, in the case of the femur B2, the CPU of the central control unit 201 superimposes the second augmented reality image AR2 that virtually represents the direction and position of the osteotomy of the femur B2 in the captured image F on the captured image F. It is displayed on the display panel 206a (see FIG. 10A).

Further, the CPU of the central control unit 201 determines the posture (preferably the posture and position) of the bone B (for example, the femoral bone B2, etc.) after one bone processing in the captured image F estimated by the estimation process. Based on the bone processing information acquired by the second acquisition process, at least one of the orientations and positions of the other bone processing is applied to the captured image F sequentially acquired by the captured image acquisition process using the augmented reality technology. The third augmented reality image AR3, which is virtually represented, is superimposed and displayed on the display panel 206a. More specifically, the CPU of the central control unit 201 virtually represents the direction and position of another bone processing with respect to the bone B after one bone processing in the captured image F specified by the second specific processing. 3 The augmented reality image AR3 is superimposed on the captured image F and displayed on the display panel 206a. For example, when the osteotomy of the head of the femur B2 is performed as one bone processing, the CPU of the central control unit 201 is used to attach the femur side member to the femur B2 after the osteotomy in the captured image F. The third augmented reality image AR3, which virtually represents the direction and position of the reaming of the femur B2, is superimposed on the captured image F and displayed on the display panel 206a (see FIG. 10B).

Further, in the case of the pelvis, the processing contents are substantially the same, and detailed description thereof will be omitted here. A second augmented reality image (not shown) that is virtually represented is superimposed on the captured image F and displayed on the display panel 206a.

As the second augmented reality image AR2, a planar one representing the osteotomy surface is exemplified (see FIG. 10A), and as the third augmented reality image AR3, a long cylindrical one is exemplified (FIG. 10). 10 (b)), which is an example and is not limited to these, and the shape and dimensions can be arbitrarily changed as appropriate. Further, the second augmented reality image AR2 and the third augmented reality image AR2 are in a semi-transparent state or a transparent state in which only the outer shape is identifiable, as in the first augmented reality image AR1. preferable.

In addition, in FIGS. 10 (a) and 10 (b), the part other than the femur B2 of the patient P is not shown, but in reality, the tissue around the femur B2 is also an image pickup unit. It is in a state of being imaged by 204 and displayed on the display panel 206a.

<Second intraoperative support processing>
The second intraoperative support process in total hip arthroplasty will be described below with reference to FIGS. 9 and 10.
FIG. 9 is a flowchart showing an example of an operation related to the second intraoperative support process using the second surgery support terminal 2A. Further, FIGS. 10A and 10B are diagrams schematically showing an example of a display screen displayed on the display panel 206a of the second surgery support terminal 2A in the second intraoperative support process.

As shown in FIG. 9, first, the CPU of the central control unit 201 includes a reference image showing the three-dimensional shape of the pelvis and the femur B2 to which the artificial hip joint of the patient P is to be attached, and preoperatively including bone processing information. Acquire the plan data D (step S11; second acquisition process).
Specifically, the CPU of the central control unit 201 reads out the tomographic image data I from the storage unit 203 and acquires it according to the second acquisition program 203f, and a reference image showing the three-dimensional shape of the pelvis and the femur B2 of the patient P. Image data of is generated and acquired. Further, the CPU of the central control unit 201 is used for the direction and position of the bone processing of the pelvis and the femur B2, and other bone processing for attaching the femur side member to the femur B2 after the bone processing. Preoperative planning data D including bone processing information related to orientation and position is acquired.

Next, the CPU of the central control unit 201 supports one of the pelvis and the femur B2 designated based on a predetermined operation of the operation unit 207 by the user (for example, the femur B2) during the second operation. It is designated as the target bone to be processed (step S12).

Subsequently, the CPU of the central control unit 201 sequentially acquires the captured image F in which the surface of the target bone of the patient P is imaged, substantially in the same manner as in step S3 of the first intraoperative support process of the first embodiment (step S13; The posture of the target bone of the patient P in the captured image F is estimated (step S14; estimation process) in substantially the same manner as in step S4 of the first intraoperative support process of the first intraoperative support process of the captured image acquisition process).

Next, the CPU of the central control unit 201 specifies the direction and position of one bone processing with respect to the target bone of the patient P in the captured image F (step S15; second specifying process).
Specifically, the CPU of the central control unit 201 has the posture of the target bone (for example, femoral bone B2, etc.) in the captured image F estimated by the estimation process according to the second specific program 203 g, and the second acquisition process. Based on the preoperative planning data D including the bone processing information acquired by, the coordinate system of the target bone in the captured image F and the coordinate system of the three-dimensional model of the target bone are aligned, and the captured image F is performed. The orientation and position of one bone process (eg, osteotomy of the femoral bone B2) with respect to the target bone in the coordinate system of the target bone within is specified.

Subsequently, the CPU of the central control unit 201 superimposes the second augmented reality image AR2 that virtually represents the direction and position of one bone processing with respect to the target bone on the captured image F by using the augmented reality technology, and displays the panel 206a. (Step S16; second display control process).
Specifically, the CPU of the central control unit 201 is one with respect to the target bone (for example, femur B2, etc.) of the patient P in the captured image F specified by the second specific processing according to the second display control program 203h. The second augmented reality image AR2 that virtually represents the direction and position of the bone processing is superimposed on the captured image F and displayed on the display panel 206a. For example, when the target bone is the femur B2, the CPU of the central control unit 201 captures a second augmented reality image AR2 that virtually represents the direction and position of the osteotomy of the femur B2 in the captured image F. It is superimposed on F and displayed on the display panel 206a (see FIG. 10A).
This allows the doctor to visually recognize the second augmented reality image AR2 and grasp the direction and position of the osteotomy of the head of the femur B2.

After that, the CPU of the central control unit 201 determines, for example, based on the preoperative planning data D, whether or not other bone processing for attaching the implant to the target bone of the patient P is necessary (step S17). ).
Here, when it is determined that other bone processing is necessary (step S17; YES), the CPU of the central control unit 201 captures the surface of the target bone after one bone processing in substantially the same manner as in step S13. The captured images F are sequentially acquired (step S18; captured image acquisition process). Subsequently, the CPU of the central control unit 201 estimates the posture of the target bone after processing one bone in the captured image F in substantially the same manner as in step S14 (step S19; estimation process).

After that, the CPU of the central control unit 201 specifies the direction and position of the other bone processing with respect to the target bone after one bone processing in the captured image F (step S20; second specifying process).
Specifically, the CPU of the central control unit 201 uses the posture of one bone-processed target bone (for example, femoral bone B2, etc.) in the captured image F estimated by the estimation process according to the second specific program 203g. Based on the preoperative planning data D including the bone processing information acquired by the second acquisition process, the coordinate system of one bone-processed target bone in the captured image F and the three-dimensional model of the target bone. By aligning the coordinate system, the orientation and position of another bone processing (for example, reaming of the femoral bone B2) with respect to the target bone in the coordinate system of the target bone after one bone processing in the captured image F can be determined. Identify.

Subsequently, the CPU of the central control unit 201 displays a third augmented reality image AR3 on the captured image F, which virtually represents the direction and position of another bone process with respect to the target bone after one bone process using augmented reality technology. The images are superimposed and displayed on the display panel 206a (step S21; second display control process).
Specifically, the CPU of the central control unit 201 has a target bone (for example, femur B2, etc.) after one bone processing in the captured image F specified by the second specific processing according to the second display control program 203h. The third augmented reality image AR3, which virtually represents the direction and position of the other bone processing with respect to the image F, is superimposed on the captured image F and displayed on the display panel 206a. For example, when the target bone is the femur B2, the CPU of the central control unit 201 captures a third augmented reality image AR3 that virtually represents the direction and position of the reaming of the femur B2 after osteotomy in the captured image F. It is superimposed on the image F and displayed on the display panel 206a (see FIG. 10B).
This allows the doctor to visually recognize the third augmented reality image AR3 and grasp the direction and position of the reaming of the femur B2.

After that, the CPU of the central control unit 201 determines whether or not there is another bone B as a target bone among the pelvis and the femur B2 (step S22). Further, even when it is determined in step S17 that other bone processing is not necessary (step S17; NO), the CPU of the central control unit 201 shifts the processing to step S22 and the target bone B Determines if there is another.

When it is determined in step S22 that there is another bone B to be the target bone (step S22; YES), the CPU of the central control unit 201 is designated as the previous target bone among the pelvis and the femur B2. After designating the other bone B (for example, the pelvis) that is not present as a new target bone (step S23), the process returns to step S13.

Then, the CPU of the central control unit 201 performs each process of steps S13 to S16 in substantially the same manner as described above, and in the second display control process of step S16, the target bone of the patient P in the captured image F (for example, for example. The second augmented reality image AR2, which virtually represents the direction and position of one bone processing with respect to the pelvis, etc.), is superimposed on the captured image F and displayed on the display panel 206a. For example, when the target bone is the pelvis, the CPU of the central control unit 201 superimposes the second augmented reality image AR2 that virtually represents the direction and position of the reaming of the acetabulum of the pelvis in the captured image F on the captured image F. Is displayed on the display panel 206a.
This allows the doctor to visually recognize the second augmented reality image AR2 and grasp the direction and position of the reaming of the pelvic acetabulum.

Then, in step S17, the CPU of the central control unit 201 determines whether or not other bone processing is required to attach the implant to the target bone of the patient P. For example, when the target bone is the pelvis, other bone processing other than reaming of the acetabulum is unnecessary, so that the CPU of the central control unit 201 determines that other bone processing is not necessary (step S17; NO). ), The process proceeds to step S22.

On the other hand, if it is determined in step S22 that there is no other bone B to be the target bone (step S22; NO), the CPU of the central control unit 201 ends the second intraoperative support process.

As described above, according to the operation support system 100 of the second embodiment, the second operation support terminal 2A has an image of the surface of the bone B of the patient P to be processed during the operation of the artificial hip joint replacement. The captured image F is sequentially acquired, the posture of the bone B in the captured image F is estimated, and the posture of the bone B in the estimated captured image F and the three-dimensional model based on the reference image of the bone B are displayed. A second augmented reality image that virtually represents the direction and position of one bone processing in the sequentially acquired image F based on the bone processing information including the direction and position of one bone processing with respect to the bone B in. Since AR2 is superimposed and displayed on the display unit 206, the bone B can be simply viewed by the doctor on the second augmented reality image AR2 displayed on the display unit 206 without using an expensive navigation system as in the conventional case. It is possible to properly grasp the direction and position of one bone processing with respect to the bone, and it is possible to provide intraoperative support for surgical operation with high accuracy and at low cost. That is, the doctor determines the direction and position of one bone processing with respect to the bone B in consideration of the direction and position of the bone B (particularly, rotation, etc.) that changes according to the state of the patient P's position on the operating table T. It can be grasped sequentially, and as a result, one bone processing can be performed more accurately.

Further, after one bone processing, the captured image F in which the bone B after the one bone processing is imaged is sequentially acquired, and one bone processing is performed virtually with respect to the sequentially acquired captured image F and the bone B. By collating with the reference image related to the three-dimensional shape in the state where the above is applied, the posture of the bone B after one bone processing in the captured image F is estimated, and one in the estimated captured image F. Includes the posture of bone B after bone processing and the orientation and position of other bone processing for attachment of the implant to the bone B after one bone processing on a three-dimensional model based on the reference image of bone B. Based on the bone processing information, the third augmented reality image AR3 that virtually represents the direction and position of other bone processing is superimposed on the image F that is sequentially acquired and displayed on the display unit 206, so that the bone B is displayed. On the other hand, even in the state where one bone processing is performed, the direction of other bone processing with respect to the bone B after one bone processing can be obtained simply by visually recognizing the third augmented reality image AR3 displayed on the display unit 206. The position can be grasped properly.

The present invention is not limited to the first and second embodiments, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, an artificial knee joint and an artificial hip joint have been exemplified as implants, but the present invention is not limited to this, and for example, a spinal cage or an artificial vertebral body can be arbitrarily changed. Further, for example, in total knee arthroplasty, the tibia B1a and the femur B1b are treated as in the second embodiment without using the intramedullary rod and the osteotomy guide member (jig) as in the first embodiment. The direction and position of bone processing may be virtually represented using augmented reality technology. At this time, when the bone processing on the bone B is performed a plurality of times, the posture of the bone B after one bone processing is estimated and the direction and position of the other bone processing are augmented reality technology as in the second embodiment. May be used to virtually represent it.

Further, in the first embodiment, as the first augmented reality image AR1, an example is shown in which both the orientation and the position of the jig such as the tibial bone marrow intramedullary rod and the femoral intramedullary rod are virtually represented. However, the present invention is not limited to this, and for example, any one of the orientation and the position of the jig may be virtually represented. Further, the first augmented reality image AR1 is changed to one that directly represents the orientation and position of the jig, or in addition, indirectly represents the orientation and position of the jig, and the orientation and position of the jig. It may represent a position that serves as a reference for adjustment (for example, in the case of tibia B1a, the posterior cruciate ligament attachment portion, the inner edge of the tibial nodule, etc.).

Further, in the second embodiment, as the second augmented reality image AR2, an example that virtually represents both the direction and the position of one bone processing is exemplified, but it is an example and is not limited to this. However, for example, one of the directions and positions of the one bone processing may be virtually represented. Similarly, the third augmented reality image AR3 exemplifies, but is not limited to, an example that virtually represents both the orientation and position of other bone processes for implant attachment. For example, one of the directions and positions of the other bone processing may be virtually represented. Further, the second augmented reality image AR2 and the third augmented reality image AR3 are changed to those that directly represent the direction and position of one bone processing or another bone processing, or in addition, one bone processing or It indirectly represents the direction and position of other bone processing, and represents the position that is the reference for adjusting the direction and position of the one bone processing and other bone processing (for example, in the case of femur B2, the bone axis, etc.). It may be a thing.
Further, when displaying the first to third augmented reality images AR1 to AR3, the three-dimensional model of the bone B is also displayed on the display panel 206a in order to show the collation state between the captured image F and the reference image. May be.

Further, in the above-described first and second embodiments, for example, a plurality of image data representing the standard three-dimensional shape of the bone B corresponding to each of the physical characteristics of a person such as race, age and gender are prepared in advance. However, if there is no tomographic image data I of the bone B of the patient P, image data matching the physical characteristics of the patient P may be used as a reference image.

Further, in the first and second embodiments, as the surgery support system 100, a configuration in which the hospital terminal 1 and the first surgery support terminal 2 and the second surgery support terminal 2A are connected via a wireless communication line is exemplified. However, this is only an example, and the present invention is not limited to this. For example, a wired cable (not shown) may be used to enable information communication. Further, the wireless LAN is exemplified as the wireless communication line, but the present invention is not limited to this, and Bluetooth (registered trademark) may be used, for example.

Further, the configurations of the first surgery support terminal 2 and the second surgery support terminal 2A are merely examples and are not limited thereto. For example, the preoperative planning process is performed on the hospital terminal 1, but this is only an example and is not limited to this. For example, the first surgery support terminal 2 and the second surgery support terminal 2A may perform the preoperative planning process. In this case, at least the tomographic image data I may be acquired from the tomographic image data I and the preoperative planning data D. Further, the first surgery support terminal 2 and the second surgery support terminal 2A may acquire the tomographic image data I and the preoperative planning data D from the hospital terminal 1 via a storage medium such as a memory card. Further, as the first surgery support terminal 2 and the second surgery support terminal 2A, for example, a wearable computer device, particularly a head-worn computer device (head mounted display) may be used.

Further, in the first surgery support terminal 2 of the first embodiment, the functions as the first acquisition means, the captured image acquisition means, the estimation means, the first display control means, and the like are predetermined by the CPU of the central control unit 201. Although it is realized by executing the program of the above, it may be realized by a predetermined logic circuit.
Similarly, in the second surgery support terminal 2A of the second embodiment, the functions as the second acquisition means, the captured image acquisition means, the estimation means, the second display control means, and the like are performed by the CPU of the central control unit 201. Although it is realized by executing a predetermined program or the like, it may be realized by a predetermined logic circuit.

Further, as a computer-readable medium containing a program for executing each of the above processes, a non-volatile memory such as a flash memory and a portable storage medium such as a CD-ROM shall be applied in addition to a ROM and a hard disk. Is also possible. A carrier wave is also applied as a medium for providing program data via a predetermined communication line.

In addition, the embodiments disclosed this time should be considered to be exemplary and not restrictive in all respects. The scope of the present invention is shown by the scope of claims rather than the above description, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

100 Surgery support system 1 Hospital terminal 2 1st surgery support terminal 2A 2nd surgery support terminal 201 Central control unit (1st acquisition means, captured image acquisition means, estimation means, 1st display control means, 1st specific means, 2nd Acquisition means, second display control means, second specific means)
203 Storage unit 204 Imaging unit 206 Display unit 206a Display panel AR1 to AR3 1st to 3rd augmented reality images B bone B1a tibia B1b, B2 femur F captured image

Claims (10)

It is a surgical support terminal equipped with an imaging unit, a display unit, and a posture sensor, and is used when attaching an implant to the femur or tibia.
To guide and support the attachment of the implant to the bone on the reference image relating to the three-dimensional shape of the femur or the tibia to which the patient's implant is to be attached and the three-dimensional model based on the reference image. The first acquisition means for acquiring the jig information including at least one of the orientation and the position of the jig of
An image capture image acquisition means for sequentially acquiring an image captured on the surface of the bone of a patient imaged by the image pickup unit.
When the distal end side of the femur is rotated with the surgical support terminal fixed at a predetermined position of the femur, the center of the femur head is based on the measurement information obtained from the posture sensor. A means for calculating the center of the femoral head, which calculates the absolute coordinates of the position of
The position of the center of the femoral head in the coordinate system of the captured image is specified from the absolute coordinates of the position of the center of the head calculated by the means for calculating the center of the head, and the captured image sequentially acquired by the captured image acquisition means. And the estimation means for estimating the posture of the bone in the captured image by collating with the reference image acquired by the first acquisition means.
Based on the posture of the bone in the captured image estimated by the estimation means and the jig information acquired by the first acquisition means, the captured image sequentially acquired by the captured image acquisition means is obtained. A first display control means for superimposing a first augmented reality image that virtually represents at least one of the orientation and position of the jig and displaying it on the display unit.
A surgical support terminal characterized by being equipped with. The orientation of the jig with respect to the bone in the captured image based on the posture of the bone in the captured image estimated by the estimation means and the jig information acquired by the first acquisition means. And further provided with a first identifying means of identifying at least one of the locations,
The first display control means superimposes the first augmented reality image that virtually represents at least one of the orientations and positions of the jig specified by the first specific means on the captured image, and displays the display. The surgical support terminal according to claim 1, wherein the surgical support terminal is displayed on a unit. The implant comprises an artificial joint component corresponding to one of the femur and tibia constituting the knee joint.
The jig comprises an intramedullary rod that is inserted into the one bone from its articular surface side.
The first display control means virtually represents at least one of the orientation of the intramedullary rod and the position of the insertion point of the intramedullary rod specified by the first specific means. The surgical support terminal according to claim 2, wherein the image is superimposed on the captured image and displayed on the display unit. The implant comprises an artificial joint component that corresponds to one of the femurs and tibias constituting the knee joint and is attached after osteotomy of the one bone.
The jig is attached to the one bone and includes an osteotomy guide member for guiding the osteotomy of the one bone.
The first display control means superimposes the first augmented reality image that virtually represents at least one of the directions and positions of the osteotomy guide member specified by the first specific means on the captured image. The surgical support terminal according to claim 2, wherein the display is displayed on the display unit. It is a surgical support terminal equipped with an imaging unit, a display unit, and a posture sensor, and is used when performing bone processing on the femur or tibia.
A reference image relating to the three-dimensional shape of the femur or the tibia to be processed in the patient's bone, and the direction and position of one bone processing with respect to the bone on the three-dimensional model based on the reference image. A second acquisition means for acquiring bone processing information including at least one of them,
An image capture image acquisition means for sequentially acquiring an image captured on the surface of the bone of a patient imaged by the image pickup unit.
When the distal end side of the femur is rotated with the surgical support terminal fixed at a predetermined position of the femur, the center of the femur head is based on the measurement information obtained from the posture sensor. A means for calculating the center of the femoral head, which calculates the absolute coordinates of the position of
The position of the center of the femoral head in the coordinate system of the captured image is specified from the absolute coordinates of the position of the center of the head calculated by the means for calculating the center of the head, and the captured image sequentially acquired by the captured image acquisition means. And the estimation means for estimating the posture of the bone in the captured image by collating with the reference image acquired by the second acquisition means.
Based on the posture of the bone in the captured image estimated by the estimation means and the bone processing information acquired by the second acquisition means, the captured image sequentially acquired by the captured image acquisition means is obtained. A second display control means for superimposing a second augmented reality image that virtually represents at least one of the directions and positions of the one bone processing and displaying it on the display unit.
A surgical support terminal characterized by being equipped with. The one bone processing on the bone in the captured image based on the posture of the bone in the captured image estimated by the estimation means and the bone processing information acquired by the second acquisition means. Further provided with a second specifying means for specifying at least one of the orientation and position of the
The second display control means superimposes the second augmented reality image, which virtually represents at least one of the directions and positions of the one bone processing specified by the second specific means, on the captured image. The surgical support terminal according to claim 5, wherein the display is displayed on the display unit. The reference image includes a reference image relating to a three-dimensional shape in a state in which the one bone processing is virtually applied to the bone based on the bone processing information acquired by the second acquisition means.
The captured image acquisition means sequentially acquires an captured image in which the bone after the one bone processing is captured.
The estimation means collates the captured images sequentially acquired by the captured image acquisition means with the reference image acquired by the second acquisition means, and the bone after the one bone processing in the captured image is collated. The surgical support terminal according to claim 5 or 6, wherein the posture is estimated. The bone processing information includes at least one of the orientations and positions of the other bone processing for attachment of the implant to the bone after the one bone processing on the three-dimensional model based on the reference image.
The second display control means further includes the posture of the bone after the one bone processing in the captured image estimated by the estimation means and the bone processing information acquired by the second acquisition means. Based on this, a third augmented reality image that virtually represents at least one of the directions and positions of the other bone processing is superimposed on the captured images sequentially acquired by the captured image acquisition means and displayed on the display unit. The surgical support terminal according to claim 7, wherein the surgical support terminal is characterized in that. A computer for a surgical support terminal equipped with an imaging unit, a display unit, and a posture sensor, which is used when attaching an implant to the femur or tibia .
To guide and support the attachment of the implant to the bone on the reference image relating to the three-dimensional shape of the femur or the tibia to which the patient's implant is to be attached and the three-dimensional model based on the reference image. First acquisition means for acquiring jig information including at least one of the orientation and position of the jig,
An image capture image acquisition means for sequentially acquiring an image captured on the surface of the bone of a patient imaged by the image pickup unit.
When the distal end side of the femur is rotated with the surgical support terminal fixed at a predetermined position of the femur, the center of the femur head is based on the measurement information obtained from the posture sensor. A means for calculating the center of the femoral head, which calculates the absolute coordinates of the position of
The position of the center of the femoral head in the coordinate system of the captured image is specified from the absolute coordinates of the position of the center of the head calculated by the means for calculating the center of the head, and the captured image sequentially acquired by the captured image acquisition means. And the estimation means for estimating the posture of the bone in the captured image by collating with the reference image acquired by the first acquisition means.
Based on the posture of the bone in the captured image estimated by the estimation means and the jig information acquired by the first acquisition means, the captured image sequentially acquired by the captured image acquisition means is obtained. A first display control means for superimposing a first augmented reality image that virtually represents at least one of the orientation and position of the jig and displaying it on the display unit.
A program characterized by functioning as. A computer of a surgical support terminal equipped with an imaging unit, a display unit, and a posture sensor, which is used when processing the bones of the femur or tibia .
A reference image relating to the three-dimensional shape of the femur and the tibia to be processed in the patient's bone, and the direction and position of one bone processing with respect to the bone on the three-dimensional model based on the reference image. A second acquisition means for acquiring bone processing information including at least one of them,
An image capture image acquisition means for sequentially acquiring an image captured on the surface of the bone of a patient imaged by the image pickup unit.
When the distal end side of the femur is rotated with the surgical support terminal fixed at a predetermined position of the femur, the center of the femur head is based on the measurement information obtained from the posture sensor. A means for calculating the center of the femoral head, which calculates the absolute coordinates of the position of
The position of the center of the femoral head in the coordinate system of the captured image is specified from the absolute coordinates of the position of the center of the head calculated by the means for calculating the center of the head, and the captured image sequentially acquired by the captured image acquisition means. And the estimation means for estimating the posture of the bone in the captured image by collating with the reference image acquired by the second acquisition means.
Based on the posture of the bone in the captured image estimated by the estimation means and the bone processing information acquired by the second acquisition means, the captured image sequentially acquired by the captured image acquisition means is obtained. A second display control means for superimposing a second augmented reality image that virtually represents at least one of the directions and positions of the one bone processing and displaying it on the display unit.
A program characterized by functioning as.

Images