JP2016158911A - Surgical operation method using image display device, and device using in surgical operation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a surgical operation system having an image processor capable of displaying a superimposition image of a patient image and necessary information in the whole surgical operations.SOLUTION: In the surgical operation system, a motion capture device having an imaging part for emitting infrared rays to prescribed positions of an operation room and receiving reflected light from marker groups arranged in prescribed intervals is arranged at least three places, and an image display device for displaying an affected part of a patient undergoing a surgery by superimposing an image obtained by a CT, an MRI, or the like on the affected part is used. The surgical operation system includes an image display device for accurately changing coordinate positions of the marker groups measured by the motion capture devices and the image displayed on the affected part of the patient by superimposing the image on the affected part in accordance with rotation amounts of the marker groups.SELECTED DRAWING: Figure 30

Description

  The present invention relates to a surgical operation method using an image display device that superimposes and displays an image obtained by CT, MRI or the like on an affected part of a patient undergoing surgery, and an apparatus used for the surgical operation.

  The principle of the optical motion capture system was proposed by Rashid et al. In 1980. This system attaches a marker to each part of the body, shoots the position with multiple cameras, estimates the 3D position by the principle of stereo (trigonometry), the current optical motion capture system, Basically the same method is used.

  The main commercial motion capture systems currently on the market use multiple infrared cameras and infrared projectors, and people (actors) attach markers made of reflective material to their bodies. On the infrared camera, an image of the marker reflecting the light of the infrared projector is projected. The projected images of the markers obtained by each camera are associated between the cameras, and the three-dimensional position of each marker is obtained by the stereo method. By mapping (inverse kinematics) from the obtained marker group to a pre-defined multi-joint model of the human body, the position, joint angle, etc. of the person can be obtained.

  The advantages of this method are that, firstly, the marker itself is small and light, and there is no connection object such as a wire, so the movement of the actor is relatively free. Second, it is high by changing the shutter speed of the camera. It is possible to perform measurement at a frequency, and thirdly, the position detection accuracy is high.

  Applications include the creation of 3D avatars for PC games and special effects movies, 3D-CAD-designed products for large-screen TVs and projectors, and a head-mounted display (HMD) as shown in Patent Document 1. There are prototype-less systems that can be used for verification and evaluation as full-scale images. Furthermore, it is used for physical ability measurement in the sports field as shown in Patent Document 2, and rehabilitation support measurement for patients in the medical field as shown in Patent Document 3.

  Motion capture technology is often used for 3D data creation that can be verified or modified in the development / research fields as described above, but is not so often used in the field where failure is not allowed. This most extreme field is surgery among medical treatments.

  If the orientation (finding the affected area) is a simple operation, it is performed with the naked eye (or loupe), and motion capture technology is unnecessary. However, as the surgical object becomes deeper in the body and the difficulty of orientation increases, a detailed surgical plan is often made based on CT and MRI data.

In the case of such a surgical operation, the procedure is as follows.
(1) CT and MRI of the affected area of the patient are imaged, and an operation plan that is considered to bring the best results to the patient is created based on these images.
(2) An incision is made at a place determined by the surgical plan (in the case of an endoscope, an endoscope insertion portion and a robot hand insertion portion),
(3) As a work to be performed with a wide field of view, treatment is performed on bones and organs before reaching the affected area to expose the affected area (in the case of an endoscope, gas is inflated to expand the inside, and each insertion part is moved. Treat it so that it is easy, and align (align) the tip (operating part) of the surgical instrument in the insertion part with the affected part.
(4) A Galileo binocular loupe (magnification 1.5 to 3) or Kepler binocular loupe (magnification 4 to 6) that allows stereoscopic viewing instead of viewing the affected area with the naked eye as shown in FIG. And attached to the glasses-type fixing part 112 for use. For a finer procedure, the affected part is enlarged using the surgical microscope 38 (magnification is 3 to 18) in FIG. 31 (c), and treatment is performed by stereoscopic viewing from the eyepiece part of the surgical microscope 38. .

  In addition, treatment is often performed while confirming data such as CT and MRI as shown in FIG. 33A on the PC monitor 111 in FIG. 31A even during the operation. Furthermore, in such advanced surgery, the three-dimensional position of the affected part of the patient and the surgical instrument is measured in real time, and this is superimposed on the CT and MRI images taken before the operation. There is an increasing demand for ensuring certainty. In particular, such trials have begun in the fields of neurosurgery, head and neck surgery, and orthopedic surgery, and the implementation of the motion capture technology will be necessary to realize this.

  However, as described above, advanced surgery requires a means for viewing the affected area in a three-dimensional manner, and it is not easy to superimpose and display this and CT or MRI images. As a means for solving this problem, as described in Patent Document 4, a head-mounted display is used, the affected part is also captured as an image, and the captured images of CT and MRI are superimposed on each other and displayed in front of the eyes. A method has been devised.

  When motion capture technology using markers is used in this field, in order to display the position of the marker on a CT or MRI image, it is necessary to perform imaging with a marker in advance during CT or MRI imaging. Sometimes it is necessary to reattach the marker at the same position as when shooting and measure the 3D position in real time. However, since the work is time-consuming, markers are set as image data on the patient's feature points for the images taken by CT or MRI, and the registration of registration of the patient and the image position before the operation (registration) May be performed.

  FIG. 31 (a) and FIGS. 32 (a) to (c) show specific methods. In FIG. 31A, the three-dimensional measuring device 107 irradiates the emitted laser light to the marker group 105 fixed to the head of the patient 100, and measures the reflected light position displacement amount from the marker group 105, whereby the marker This is a system that can accurately measure the world coordinate position of the group 105. In addition, a surgical recording video camera 110 for imaging the head of the patient 100 from above is installed, and the image is displayed on the PC monitor 111. The marker group is defined as a group of a plurality of markers used for arranging a plurality of markers in a predetermined arrangement and registering their shapes.

  FIG. 32A shows a state when performing surgery on the head of the patient 100. In the operation of the head of the patient 100, the incision is reached after the incision. Therefore, it is necessary to detach the skull and muscles or to make a hole in the skull, and a considerable force is applied to the patient. Therefore, the head of the patient 100 is firmly fixed to the operating table 109 by the three-point fixture 104. On the other hand, before the operation, an MRI image 101 as shown in FIG. The MRI 113 stores the internal video as image data with the human body in a circular shape. Therefore, the surface information of the human body can be obtained by performing image processing that combines the images of the outer periphery, thereby obtaining a three-dimensional surface 3D image 102 of the human body as shown in FIG.

  At this time, the head of the patient 100 during the operation in FIG. 32A and the surface 3D image 102 in FIG. 32B created by the MRI image are not related. In order to superimpose and display the two, first, the virtual marker 103 is set on the surface 3D image 102 of FIG. Since the virtual marker 103 is installed by sequentially specifying the feature points 108 of the head of the patient 100 in FIG. 32A using the calibrator 106, a characteristic earlobe or nasal root is selected.

  Although it is the calibrator 106, it is comprised by the front-end | tip part a and the marker group part b which perform a pointed tentacle, The relative positional relationship is measured beforehand. Since the head of the patient 100 is firmly fixed by the three-point fixture 104, it is assumed that the patient group 100 has a predetermined positional relationship with the marker group 105 installed on the three-point fixture 104.

  As shown in FIG. 32 (a), the feature point 108 of the head of the patient 100 is sequentially touched with the tip of the calibrator 106, and the world coordinate position of the marker group part b at that time (the world coordinate position is a motion capture device). (Which indicates the absolute coordinate system within the measurable area) determined by (1). Since the world coordinate positions of the marker group 105 and the marker group part b are measured by the three-dimensional measuring device 107, local coordinates of the calibrator 106 (locus coordinates indicate relative coordinates at two predetermined locations. One world coordinate Is subtracted from each world coordinate position, the world coordinate position of the feature point 108 is determined.

  Since the feature points 108 respectively correspond to the virtual marker group 102, when the affected part of the actual patient shown in FIG. 33 (b) is brought into contact with the tip portion b of the calibrator 106, the MRI shown in FIG. 33 (a). The position is displayed as a white circle on the output image. In this way, initialization registration work (registration) is a process for determining the coordinate axis conversion parameters for those having different coordinate axes so that they can be superimposed and displayed when both images are combined.

  However, when moving the position of the head of the patient 100, accurate tracking (here, tracking control is performed so that each image is superimposed and displayed according to the movement of the object with the marker attached thereto). In addition to the marker fixing method and the problem of reproducibility of the fixed position, as described above, initialization registration work (registration) of the patient and the image position requires complicated adjustment work. The actual tracking accuracy depends on the accuracy of the sensor itself, and the tracking accuracy when performing treatment on the enlarged image requires high accuracy in proportion to the enlargement rate of the affected area image.

  There are still many problems, but in addition, there are the following advantages if CT and MRI images can be superimposed and displayed on the affected part at each stage of surgery. . It is thought that the merits also occur in the following intraoperative work by the superimposed display.

  In the stage (2) described in the prior art, the location of the incision is actually written on the patient's skin. However, if the mark can be superimposed and displayed on the patient as an image, the writing operation becomes unnecessary.

  In the stage (3) described in the prior art, when an endoscope is used, the field of view of the endoscope is narrow, and therefore an operation of aligning the distal end with the affected part is required. In order to move the endoscope freely and access the affected area, it is necessary to inject gas and create a cavity between the affected area and the skin. At this time, since the organ moves in the cavity, the position of the organ taken in by CT or MRI often differs from the actual intraoperative position, and it takes time to specify the position. If there is a method that can superimpose and display the position of the tip and the affected part on the patient as an image, the alignment work is greatly shortened.

  In the stage of (4) described in the prior art, it is sufficient if the CT or MRI image taken in (1) can be accurately superimposed and displayed on the affected part as necessary. However, in order to prevent infection as shown in FIG. Cover the patient with a sterile cover 31 except for the affected area, and a transparent sheet (film roll with excellent oxygen and water vapor permeability, which prevents water, bacteria and viruses from passing through and prevents contamination from the outside. Covers the skin as a frictional measure that causes pressure ulcers. The affected part is covered with 32). The marker 105 fixed to the three-point fixing tool 104 during the operation can be observed through the transparent drape, but the feature point 108 corresponding to the virtual marker group 102 on the head cannot be seen, so that re-registration cannot be performed. Since various treatments are performed during the operation, the position of the patient sometimes changes with respect to the three-point fixing device 104. In this case, the superimposed display image is shifted and cannot be used. If the superimposed display image can be registered even during the operation, the operation can be guided (navigation).

  In advanced surgery, a surgical microscope 38 for two-person observation is used. For this reason, it is desirable for two people to see a three-dimensional image of the affected part at the same time, and it is desirable to see the superimposed display image at the same time, but the mechanism is complicated in structure. Patent Document 5 proposes to improve the complicated mechanism, but the technology for viewing the PC monitor 111 in 3D itself is not consistent with the actual appearance. When changed, there was no simple method that could perform overlay display with high accuracy.

JP 2013-218535 A JP2013-192591A International Publication No. 2005/096939 Japanese Patent No. 5390377 JP 2006-218206 A

  As described above, there are not so many proposals for a surgical system and an apparatus constituting the system capable of accurately and efficiently displaying a necessary superimposed image in all of these surgical processes.

  The present invention has been made in view of the above problems, and measures the three-dimensional positions of a surgical instrument and a patient who is operating at the time of surgery in real time, and CT or MRI taken before surgery based on this result. Surgical system having an image processing apparatus that can easily superimpose and display the image of the patient on the affected area, and can superimpose and display the patient or patient image and necessary information at all surgical stages. The purpose is to provide.

  In a first embodiment for solving the above-described problem, a motion capture device having an imaging unit that emits infrared rays to a predetermined position in an operating room and receives reflected light from a marker group arranged at a predetermined interval. According to the coordinate position and rotation amount of the marker group measured by the motion capture device, the image display device is arranged at least in three places and has an image display device that superimposes and displays a predetermined first image on the affected part of the patient undergoing surgery. In a surgical operation system having an image display device that changes a superimposed display image with an affected part of a patient, the image display device includes a camera part that takes an image of the affected part, a depth camera part that measures the depth of the affected part, and further It is characterized by having one marker group.

  There should be at least two sensor units 1 for motion capture. During the operation, the first surgeon and the second surgeon, and also the supporting nurse are around the patient 100. In many cases, it stands at a position that blocks the optical path. Position measurement by such a stereo image processing method divided into a plurality of locations can accurately measure the position coordinates unless the optical paths of at least any three motion capture sensor units 1 are shielded. Compared with the method using the three-dimensional measuring device 107 in which the measurement position is limited, the intraoperative restriction is reduced, and it is not necessary to secure a space for placing the three-dimensional measuring device 107 or the like.

  Further, as shown in FIG. 2, the tablet 4 that is usually marketed has a built-in camera, and when the head of the patient 100 is taken with the built-in camera of the tablet 4, the head image of the patient 100 is displayed on the screen. The On the other hand, the MRI stores the internal video as image data with the human body cut into a circle. Therefore, the surface information of the human body can be converted into polygon data 9 by using the outer peripheral image as point cloud data. Similar to the prior art, by performing registration measurement, the head image of the patient 100 and the superimposed display image 5 based on MRI polygon data can be observed on the screen of the tablet 4.

  It is possible to display the superimposed display image 5 on the PC monitor 111. However, since the tablet 4 can be used for direct confirmation with the line of sight during the operation, the target tumor or the position of the artery or nerve where protection is required There is an effect that it is easy to grasp. Furthermore, when the tablet 4 is held by a tool or a nurse, the position of the arteries and nerves can be grasped while performing the operation on the affected area, so that the efficiency and safety of the operation can be improved. It becomes.

  Further, FIG. 4 shows an actual working scene in the case where a depth camera 6 attached to the tablet 4 is used instead of the built-in camera. Time of Flight has been proposed as the simplest and best performance method for the depth camera 6. As shown in FIG. 3, this is a method of measuring the distance to the object by blinking the LED illumination 7 with a predetermined sin curve and detecting the phase delay D of the reflected light from the object. That is, in the case of the depth camera 6 having the two-dimensional light receiving element 8 with the objective lens, it is possible to detect the phase delay for each element, and the distance to the surface of the object is detected by the fineness of the number of divided elements. it can. Originally, the two-dimensional light receiving element 8 with the lens of the depth camera 6 itself has the position information of the top, bottom, left and right, and further, by taking the depth information by Time of Flight measurement, the surface information of the three-dimensional object 100 is obtained. Can be obtained.

  The depth camera 6 will be described in more detail. Since the speed of light is 300,000 km / sec, for example, if the LED modulation frequency is 20 MHz, the light modulation wavelength is 15 m. Although it depends on the amount of noise in the CCD light receiving unit of the depth camera, the accuracy of measuring the wavelength in the range 256 to 1024 is considered to be accurate in general phase detection, so the measurement accuracy in the Z direction of the depth camera is about 15 to 60 mmZ. However, recently, a PC clock frequency of 250 MHz is available for CMOS, and if this clock can be used to modulate the LED and this also supports the CMOS sampling frequency, it can be divided into 4 parts necessary for phase detection. Therefore, one dot phase can be detected once at about 60 MHz. The light modulation wavelength is 5 m, and the shorter the wavelength, the higher the accuracy of phase detection. In the present invention, since a certain amount of time may be required for registration, high accuracy can be expected in measurement in the Z direction.

  As shown in FIGS. 32A to 32C, in the conventional technique, the finite virtual marker 102 is matched with the feature point 108 of the head of the patient 100 by eye measurement, so that the accuracy of the superimposed display is greatly deteriorated. However, the surface information of the three-dimensional human body using the depth camera 6 is measured as point cloud data as shown by 11 in FIG. 5 when viewed from the measurement direction, and is completely separated from surrounding images having different distances. It becomes point cloud data. On the other hand, the surface information of the three-dimensional human body obtained by MRI is converted like polygon data 9 when three-dimensional display is performed, but the data of MRI itself is like point cloud data 10. It consists of a stack of images. Therefore, when performing pattern matching, when using the method of extracting feature points and comparing the vertical and horizontal lengths of appearance, it is point cloud data that is completely separated from the surrounding image, and many Since points are used, highly accurate registration is possible due to highly accurate coordinate positions and a high averaging effect.

  In the second embodiment, a motion capture device having an imaging unit that emits infrared light to a predetermined position in an operating room and receives reflected light from a marker group arranged at a predetermined interval is arranged to An image display that includes an image display device that superimposes and displays a predetermined first image on the affected area, and changes the affected area of the patient and the superimposed display image according to the coordinate position and rotation amount of the marker group measured by the motion capture device In a surgical operation system having an apparatus, a first endoscope having a concave portion at a distal end portion has a first marker group, and an affected part is identified from an image of an entire organ taken by the first endoscope. And the second endoscope or surgical instrument for displaying an enlarged image of the affected area of the patient has a second marker group, and the insertion direction of the second endoscope or surgical instrument is It is characterized in that displays the status of the relative positions of the position of the distal end portion and a position of the identified affected area.

  As shown in FIG. 8A, the wide-angle laparoscope 12 according to the present invention employs a concave lens 12b in the light receiving portion at the tip, so that a wide visual field can be observed from the position of the incised hole as shown in FIG. It is a possible device. Further, as shown in FIG. 6, the thin portion to be inserted is shorter than the portion of the laparoscope 13 to the extent that the tip portion of the trocar 15 protrudes from the tip. Therefore, it can be seen that the structure is stable when inserted, and is suitable for projecting a wide field of view. FIG. 7B shows a state in which another surgical instrument is inserted while viewing the affected part with the wide visual field. Without getting lost, the tips of the laparoscope 13 and forceps 14 can be inserted to the affected area in a short time. In this state, when all the surgical instruments are inserted and aligned, the monitor image is switched to the image of the laparoscope 13 as shown in FIG. 7A, so that the alignment operation is shortened and smooth regardless of the organ position. It is possible to perform a surgical operation.

  In the third embodiment, at least three motion capture devices having imaging units that emit infrared rays at predetermined positions in the operating room and receive reflected light from marker groups arranged at predetermined intervals are arranged, and surgery is performed. An image display device that superimposes and displays a predetermined first image on the affected area of the patient who receives the image, and displays the superimposed image on the affected area of the patient according to the coordinate position and rotation amount of the marker group measured by the motion capture device. In a surgical operation system having an image display device to be changed, a medical device that performs an operation on an affected part of a patient has a first marker group that can be attached and detached, and can be supported by the action part of the medical device and at least two other points. The calibration device has a second marker group, and the relative coordinate position of the second marker group and the portion that comes into contact with the distal end portion (action portion) of the medical device are determined in advance. While 憶, wherein measuring the position coordinates of the first marker group and the second group of markers when the calibration device on the medical device is placed in the motion capture apparatus is characterized in that storing.

  Similar to the calibrator 106, if the tip portion a for performing a pointed tentacle is used as the tip (operating portion) of the surgical instrument and the marker group portion b is set and the interval thereof is measured in advance as local coordinates, a motion capture sensor By examining the position coordinates of the marker group in the section 1, the world coordinate position of the tip (operating section) of the surgical instrument can be specified. However, these surgical instruments are always sterilized and used repeatedly. On the other hand, since the marker group has an irregular structure and the marker part is difficult to remove, when attached to a surgical instrument, the marker group is likely to be disposable or removed and cleaned and sterilized separately.

  In the present invention, even if the marker group is removed from the surgical instrument for cleaning or the like, the marker group is affixed to the surgical instrument and placed on the calibration device instantly (tip) of the surgical instrument. Since the local coordinates of the marker group can be measured, it is possible not only to prevent infections and secondary infections, but also to perform efficient cleaning, respectively, and to improve the cleaning efficiency of equipment. In addition, since measurement is performed before surgery without difficult work in the operating room, there is an effect that the possibility of drift due to environmental changes and marker group wear can be reduced.

  Further, by combining the second and third modes, the following new functions can be introduced. As shown in FIG. 12, first, when the wide-angle laparoscope 12 is inserted, the whole image of the organ is confirmed to confirm where the affected part (black x mark in the figure) is. FIG. 13A shows the appearance of the image at that time. Since a circle mark 24 is displayed in advance at the center of the image, a wide-angle laparoscope is placed so that the affected area comes within the circle mark 24. The angle and position of 12 are moved, and the affected part measuring rod 12p is inserted into the hole 12c from the wide-angle laparoscope 12 until the tip reaches the affected part. After confirming that the tip has entered the circle mark 24, the world coordinate position and the image of the tip of the affected part measuring rod 12p are stored. That is, since the tip of the affected part measuring rod 12p is in contact with the affected part, the coordinate becomes the world coordinate position of the affected part.

  Although the stored image is shown in FIG. 13B, it is an already stored image, and the stored image continues to be displayed on the monitor even when the wide-angle laparoscope 12 is pulled out from the trocar 15. Next, the laparoscope 13 is inserted. Since the world coordinate position of the distal end portion (light receiving portion) of the laparoscope 13 has already been obtained, the distal end coordinate position is superimposed and displayed as a cross mark in FIG. 13B. The position of the affected part and the tip part (light receiving part) of the laparoscope 13 is also displayed as position difference information, as in A (0, 0, +5). Similarly, for example, even if the forceps 14 or the electric knife 18 is inserted from another trocar 15, the distal ends are displayed with differently marked circles 25 and Δ 26, respectively, and the affected area and the distal ends of the forceps 14 and the electric knife 18 are displayed. The position of each part is also displayed as position difference information, such as B (+30, +40, +20) and C (−20, +20, +10). When these marks are almost in the circle, if the image of FIG. 13B is switched to the image of FIG. 7A received by the laparoscope 13, all the surgical instruments inserted in only the instantaneous alignment time. Can be confirmed within the narrow field of view. Since the wide-angle laparoscope 12 can be pulled out and another surgical instrument can be inserted, the number of incisions can be reduced and a quick alignment operation can be performed.

  In the fourth embodiment, a motion capture device having an imaging unit that emits infrared rays to a predetermined position in the operating room and receives reflected light from a marker group arranged at a predetermined interval is arranged, and a patient undergoing surgery is arranged. An image display that includes an image display device that superimposes and displays a predetermined first image on the affected area, and changes the affected area of the patient and the superimposed display image according to the coordinate position and rotation amount of the marker group measured by the motion capture device In the surgical operation system having the apparatus, the image display device includes a camera that captures an image of the affected area and the first marker group, and superimposes and displays the second image and the first image captured by the camera. For this purpose, at least one or more second markers are attached to the transparent sheet pasting region covering the affected area of the patient.

  In the stage of (4) explained in the prior art, as shown in FIG. 34, in order to prevent infectious diseases, other than the affected part is covered with a sterile cover 31, and further, the affected part is covered with a transparent sheet 32, so that it is fixed to the patient during the operation. It becomes impossible to see the marked marker 105. Since various treatments are performed during the operation, the position of the patient is often changed. However, according to the present invention, the second marker is attached to the transparent sheet 32, so that the second marker can be used even during the operation. It is possible to check the position of the patient, and even if the position of the patient changes, it is possible to control so that the superimposed image display is not shifted by checking the second marker.

  For easy understanding in FIG. 14, the relationship between the size of the transparent sheet 32, the aseptic cover 31, the opening 31 a for the surgical operation, the affected part 29 (incision part), and the marker 30 with respect to the patient 100 and the three-point fixture 104. As you can see, everything is shown as a rectangle. First, before covering the patient 100 with the aseptic cover 31, registration is performed using the first and second embodiments. At this time, the superimposed display is performed without any problem. Next, the marker 30 is pasted in the vicinity of the affected part 29. Using the cameras used in the first to second embodiments, the position of the marker 30 is recognized as an image. Since the local coordinate values of the marker group 105 and the MRI superimposed image 9 have already been obtained, if the local coordinate values of the second marker 30 and the marker group 105 are obtained, the local coordinate values of the marker 30 and the MRI image 9 can also be obtained. become. That is, since the marker 30 is affixed to the head of the patient 100, the MRI image 9 and the head image of the patient 100 can be accurately superimposed and displayed by examining and correcting the local coordinate value variation of the marker 30. it can.

  In FIG. 17, the sterile cover 31 is aligned so that the affected area 29 and the marker 30 are exposed from the opening 31a, and the other patient 100, the three-point fixture 104, and the marker group 105 are completely covered with the sterile cover 31. Further, the state where the affected part and the marker 30 are covered with the transparent sheet 32 is shown. Since the marker group 105 is hidden, the position of the head of the patient 100 cannot be detected by the motion capture sensor unit 1, but even if the position of the head of the patient 100 drifts, the camera 100 is used on the transparent sheet 32. By examining and correcting fluctuations in the local coordinates of the marker 30 that can be seen through, the MRI image 9 and the head image of the patient 100 can be accurately superimposed and displayed on the monitor while reducing the risk of infection. .

  Although the AR marker is taken as an example this time, the AR marker is originally used for image recognition and the position accuracy itself is not so high. Originally, the amount of movement of the head of the patient 100 during the operation is not so large, and a large square pattern with a black frame as shown in FIG. 15B may be used instead of the AR marker. One-dimensional data X1, X2, Y1, and Y2 are formed by taking in this pattern as two-dimensional image data, dividing it like a dotted line, and performing vertical and horizontal addition processing for each divided region. By detecting the slope signal of the black frame portion and analyzing each slope position, it is possible to accurately detect not only the shift amount of x and y but also the rotation amount, pitching, and rolling error. Since the pattern is simple, it is not suitable for image recognition, but in a situation where the position of the pattern is fixed to some extent (a condition where the amount of head drift of the patient 100 is small), this pattern with high averaging effect can be obtained by performing addition processing. It is possible to maintain the superimposed display state with higher accuracy.

  Furthermore, in the fifth embodiment, at least three motion capture devices having imaging units that emit infrared rays at predetermined positions in the operating room and receive reflected light from marker groups arranged at predetermined intervals are arranged. And an image display device that superimposes and displays a predetermined first image on the affected area of the patient undergoing surgery, and superimposed and displayed on the affected area of the patient according to the coordinate position and rotation amount of the marker group measured by the motion capture device In the surgical operation system having an image display device that changes an image, the image display device includes a camera that takes an image of the affected area and the first marker group, and the second image and the first image taken by the camera. In order to superimpose and display one image, a connector part capable of fixing the second marker group is attached to a part of the patient from the affected part of the patient not through the joint part. That.

  As shown in FIG. 16, the marker group 34 is removed from the connector portion 33, covered with a sterile cover 31, and a transparent sheet 32 is pasted from above the connector portion 33. There is a flat portion 33a having a large pasting area. When the transparent sheet 32 is pasted from above the connector portion 33, a convex portion 33b hollow portion as shown by a dotted line is generated in the flat portion 33a. When the transparent sheet 32 is cut along the dotted line, the convex portion 33b is exposed. However, since the flat part 33a and the adhesive back surface of the transparent sheet 32 are firmly adhered to the dotted line 32a, if the connector part 33 and the marker group 34 are sterilized, the bacteria on the back side will be exposed on the surface during the operation. No trouble occurs.

  In FIG. 18, the aseptic cover 31 is aligned so that the affected part 29 and the connector part 33 are exposed from the opening 31a, and the other patient 100, the three-point fixture 104, and the marker group 105 are completely covered with the aseptic cover 31. In addition, after the affected part and the marker 30 are further covered with the transparent sheet 32, it is cut along a dotted line 32a including the convex part 33b portion of the connector part 33 to expose the convex part 33b. Since the marker group 105 is hidden, the head position of the patient 100 cannot be detected by the motion capture sensor unit 1, but the marker group 34 is attached to the convex portion 33b even if the position of the head of the patient 100 drifts. The MRI image 9 and the head image of the patient 100 can be accurately superimposed and displayed on the monitor by examining and correcting the fluctuation amount of the marker group 34 using the plurality of motion capture sensor units 1.

  In FIG. 19, the fifth mode is realized by another method. The vicinity of the affected area 29 is an important area at the time of surgery, and it may be difficult to attach the marker 30 and the connector section 33 there. Therefore, in FIG. 19, the head away from the affected area 29 is incised, a screw hole is dug in the skull, and the connector part 35 having a slightly higher strength is embedded. However, since the entire patient 100 is covered with the sterile cover 31, the arm part 36 that can be connected to the connector part 35 is inserted, and relayed to a position outside the sterile cover and without interfering with the operation. A marker group 37 is attached to the tip as a connector portion. The installation itself is a process that takes almost no time. During installation, the arm is supported by a person so that no load is generated on the skull part where the screw hole is dug.

  Using the image composition device, the 3D image obtained from MRI and the video data obtained from the image input unit are superimposed and displayed according to the result of the motion capture device. It is necessary to move the line of sight from the line of sight to the PC monitor 111, making it difficult to immediately grasp the location of the arteries and veins and nerves while performing the operation. In the sixth embodiment, a motion capture device having an imaging unit that emits infrared light to a predetermined position in an operating room and receives reflected light from a marker group arranged at a predetermined interval is arranged, and a patient undergoing surgery is arranged. An image display device that superimposes and displays a predetermined first image on the affected area, and changes the patient's affected area and the superimposed display image according to the coordinate position of the marker group and the rotation amount of the mark group measured by the motion capture device; In the surgical operation system having the image display device to be operated, the image display device has first and second magnifying optical systems for independently viewing the affected area with the left and right eyes, and the first and second magnifying optics. Between the system and the eyes, it has first and second composite mirrors for displaying the projected image of the affected area and the first image in a superimposed manner, and further has a first marker group. .

  FIG. 22 is a perspective view when the embodiment of the sixth embodiment is mounted, and FIG. 23 is a side view of the embodiment of the sixth embodiment. This is a structure in which a Kepler type (magnification 4 to 6) magnifying part (also a Galileo binocular loupe type) 42 and an objective lens part 44 are joined by an image synthesizing part 45. The in-focus point of the loupe unit 42, the two-dimensional image display liquid crystal unit 43 attached to the tip of the objective lens 44, and the retina of the eye are in a conjugate position, and are configured as a pair for the right eye and the left eye. A configuration in which these optical portions are held by the detachable portion 41, and the entire image output device 40 can be stably positioned at the positions of both eyes by connecting the detachable portion 41 to the spectacle-type fixing portion 112 that can be attached. It has become. A pair of built-in cameras 63 are also installed in the detachable section 41, and information obtained there can be viewed on the liquid crystal section 43 for two-dimensional image display.

  The weight of the loupe unit and the glasses-type fixing unit 112 alone is 70 to 90 g. When the objective lens unit 44 is combined with the image synthesis unit 45 and the two-dimensional image display liquid crystal unit 43, the weight is about 150 to 200 g. . It is difficult to support this beside the ears and nose as with glasses. Therefore, in this image output device 40, an elastic rod 60 made of a shape memory alloy is attached to the detachable portion 41, and the image output device 40 uses the fulcrum portion 59 protruding from the support surface 58 on the head of the head as a fulcrum. Balance is maintained by balancing the weight of the balancer 61 with a shape like "Yajirobe". The position of the elastic bar 60 and the fulcrum, and the position and weight of the balancer 61 can be finely adjusted. When the binocular loupe is worn, there is no problem that the supporting part next to the nose hurts due to its weight, and it is possible to keep wearing it stably.

  With this structure, all the weight is applied to the support surface 58 under the fulcrum 59, but the weight is applied to the entire head, so there is almost no burden on the neck. Moreover, even if the weight of the balancer is combined, it is as heavy as a construction helmet and the area of the support surface 58 is large, so that the weight is supported by the entire head. There is no discomfort due to imbalance or tightening like a binocular loupe or head-mounted display, and it can be worn for a long time even during surgery.

  Further, as shown in FIG. 23, like a normal binocular loupe, by folding the connector part, the loupe part 42 can be retracted from the line of sight of the eye as in the binocular loupe, and a wide field of view (naked eye or glasses) is obtained. ) To continue the surgery.

  Since the marker group 34 is also attached to the attachment / detachment portion 41 of the image output device 40, registration can be performed. If the image from the built-in camera 63 and the 3D image obtained from the MRI are displayed as a superimposed image on the two-dimensional image display liquid crystal unit 43 by registration, all the conditions necessary for carrying out the first and fifth embodiments are obtained. organized. Furthermore, if the built-in camera 63 is changed to the depth camera 6, the second embodiment can be implemented. By allowing the built-in camera 63 to detect the AR marker, the fourth embodiment can be implemented.

  Furthermore, if the video from the CCD camera of the laparoscope 13 and the tip information of the surgical instrument are displayed on the two-dimensional image display liquid crystal unit 43 as a superimposed image, the third embodiment can be implemented. However, in this case, the appearance of the loupe unit 42 is obstructive. As shown in FIGS. 25 and 26, the image output device 40 is provided with a shield plate 46 that can be inserted and removed between the split mirror (half mirror) 45 and the loupe unit 44. When it is desired to view only the image from the CCD camera of the laparoscope 13 as in the case of endoscopic surgery, a shielding plate 46 is inserted as shown in FIG.

  The split mirror may be a polarizing mirror, and a transmissive liquid crystal element may be placed in the 46 portion. Since the transmitted light of the polarizing mirror is P-polarized light and the image coming from the liquid crystal unit 43 for two-dimensional image display is S-polarized light, the same can be done if the P-polarized light is electrically shielded by the transmissive liquid crystal element unit. On the other hand, when only the real image of the loupe unit 42 is desired to be viewed, the image of the two-dimensional image display liquid crystal unit 43 is turned off and the light shielding plate 46 is retracted as shown in FIG. In the case of using a transmissive liquid crystal element portion, it is sufficient to electrically transmit P-polarized light.

  In addition, since the image output device 40 can observe the actual intraoperative state from the loupe unit 42, the image of the liquid crystal unit 43 for two-dimensional image display can be superimposed on the real image. For this usage, a separate registration work is required. Since the interval between the pair of loupes 42 and the interval between the built-in cameras 63 arranged in a pair are substantially equal, when these are superimposed and displayed, both images can be seen by the half mirror. When the view center of the loupe unit 42 is viewed, it is designed to be parallel to the view center of the built-in camera, and the position and magnification of the image on the liquid crystal unit 43 for two-dimensional image display are matched so that its appearance and magnification match. Adjust. If the left and right eyes match, the registration operation is complete.

  FIG. 24A shows an affected area when viewed with a binocular loupe 114 having the same magnification as the image output apparatus 40, and FIG. 24B is an MRI stereoscopic image when viewed from the line of sight. By mounting the image output device 40, as shown in (c), it is possible to accurately overlay the 3D image obtained from the loupe unit 42, the image, and the MRI. Stereoscopic viewing is possible by the loupe unit 42, and there is an advantage that the position of the tumor V, the position of the vein J, etc. can be confirmed without changing the line of sight while performing the operation. In general, when viewing an object through a lens, the pupil position of the eye always changes when looking around. As a result, the position of the chief ray passing through the lens changes, and the distortion also changes. Although it is not easy to change the superimposed image in response to this change in distortion, the present invention uses a loupe with a narrow field of view and also uses motion capture to track the image. Can be seen near the center of the loupe lens. Since there is almost no distortion in the image near the center, there is almost no need to correct the distortion of the image to be superimposed.

  The last thing to do is when using a surgical microscope for two-person observation in advanced surgery. If the first surgeon and the second surgeon can see the three-dimensional image of the affected area at the same time, and can see the superimposed display image at the same time, efficient and safe navigation is possible. In order to achieve this, in the seventh embodiment, at least a motion capture device having an imaging unit that emits infrared rays to a predetermined position in an operating room and receives reflected light from marker groups arranged at predetermined intervals is provided. In the surgical operation system having an image display device arranged at three locations and changing a display image of a diseased part of a patient according to a coordinate position of the marker group measured by the motion capture device and a rotation amount of the mark group. The display device is a head-mounted display in which a first marker group is arranged, and a two-dimensional image sensor that receives an image of the affected area is arranged on the first imaging plane of the stereoscopic microscope that images the affected area of the patient. The amount of rotation and shift of the stereomicroscope with the affected area as the center of rotation is controlled according to the movement of the face of the wearer of the head mounted display. A control mechanism is characterized by controlling the rotation and shift amount of the stereoscopic microscope and depending on the movement of the face of the wearer.

  27 (a) to 27 (b) are examples in the seventh embodiment, (a) is a front view, and (b) is a side view. When the light of the LED illumination 62 irradiates the observation target T, the diffuse reflected light from the target T becomes a parallel light flux through the deflecting lens 48 as a light flux having a predetermined inclination. Thereafter, the image of the target T is projected onto the two-dimensional CCD 51 through the zoom optical system 49 and the objective / focus adjustment lens 50. This is because the distance between the light beams in the deflection lens 48 is set to be 62 mm, which is the distance between the eyes of the person, and the stereoscopic vision when viewing the target T with both eyes positioned at the deflection lens 48 is almost the same. is there.

  That is, when the image of the two-dimensional CCD 51 is viewed on, for example, the head mounted display type image display device 40 described in the sixth embodiment, the operation is performed while looking at the target T with both eyes placed at the position of the deflection lens 48. The conditions are almost equivalent to. Since the image of the two-dimensional CCD 51 can be electrically distributed, all persons wearing a head mounted display type image display device can view a stereoscopic image under the same conditions. In general, the second surgeon often performs work on the opposite side of the operating table 109 of the first surgeon. From the viewpoint of the second surgeon, the first surgeon's line of sight is looking at an image rotated 180 degrees with respect to the normal viewing condition. This means that your right hand appears from the left side, which is very difficult to do. Since the image of the two-dimensional CCD 51 can be easily rotated 180 degrees by image processing, it is possible to increase the efficiency of the support by showing the second surgeon the image rotated 180 degrees.

  Since the image display device 40 is a head-mounted display type, an image can be viewed at an arbitrary position. However, when the surgeon performs an operation, the affected part of the patient 100 is at the position of the target T, and the fingertip needs to work at that position. Therefore, the positional relationship between the stereoscopic microscope 47 and the image display device 40 is as shown in FIGS. (A) is a front view, and (b) is a side view. Here, when the distance between the deflection lens 48 and the target T is L, the surgeon is in the most comfortable line-of-sight state (there is an individual difference, but usually 20 to 30 degrees downward), and the virtual target is L It will be a distant TT. The direction of the eyes and TT is the Z-axis direction in the image display device 40, and the angles of the X-axis and the Y-axis are determined accordingly, and this is the coordinate axis B in the virtual space.

  Since the surgeon sees the affected part of the patient 100 at the position of TT on the coordinate axis, the surgeon can control the stereoscopic microscope 47 in the same direction as the direction of the face during the operation. Since the affected part can be seen in accordance with the line of sight, the trouble of changing the viewing position and angle by moving a large stereoscopic microscope support arm each time as in the conventional surgical stereoscopic microscope is eliminated. This not only increases work efficiency and safety, but also has the effect of preventing virtual sickness.

  FIGS. 29A and 29B show how to move a normal surgeon's face. The horizontal movement around the Z axis with reference to the coordinate axis B with respect to the virtual target TT is a rotation along the paper surface of FIG. 29A and a lateral shift along the paper surface of the stereoscopic microscope 47, and therefore a rotation stage around the Z axis. The rotation around the Z axis of 53c and the X drive of the X axis moving stage 54X can be handled only. Further, the vertical movement around the X axis with reference to the coordinate axis B with respect to the virtual target TT becomes a rotation along the paper surface of FIG. 29B and a vertical shift along the paper surface of the stereoscopic microscope 47. The rotation around the X axis and the Z drive of the Z-direction moving shaft 56 can be handled. Since each drive system can be controlled independently, smooth drive control of the stereoscopic microscope 47 can be performed according to the movement of the face.

  For the movement of the face, the movement of the marker group 34 installed in the image display device 40 is measured using a plurality of motion capture sensor units 1, and the stereoscopic microscope 47 is driven and controlled according to the fluctuation position and fluctuation angle with respect to the coordinate axis B reference. Just do it. It is also possible to feedback control whether or not the stereoscopic microscope 47 is driven as expected by installing a marker group in the stereoscopic microscope 47 and measuring it using a plurality of motion capture sensor units 1. Although the image display device 40 is used this time, it goes without saying that the same thing can be achieved by using a normal surgical head mounted display 39.

  As for the superimposed display when viewing the image of the stereoscopic microscope 47 with the image display device 40, if a marker group is installed in the stereoscopic microscope 47, the two-dimensional CCD 51 originally functions as a built-in camera. Can be measured using the third to fifth aspects of the invention. However, the zoom optical system 49 is mounted on the stereoscopic microscope 47, and when the magnification of the image is changed by the zoom optical system 49, the superimposed display control using the marker group cannot be performed. In this case, by confirming the AR marker used in the fourth form arranged in the vicinity of the target T with the two-dimensional CCD 51 and correcting the magnification and the position change, stable superimposed display is possible.

  In addition, since the driving unit of the stereoscopic microscope 47 can be controlled independently without interfering with each direction, the degree to which the superimposed display image drifts is automatically measured in advance according to the variation of each encoder value. The error may be stored as a map. Regarding errors due to mechanical tolerances and deformation of the self-weight of the stereoscopic microscope 47, the drift of the superimposed display image can be minimized by correcting with a variation error map by automatic measurement in that state.

  As described above, according to the present invention, by providing the first to seventh modes when the superimposed display of images is necessary in the surgical procedures (1) to (4) described in the conventional technique, There is an effect that the display image can be provided with high accuracy.

The block diagram of the surgery system by the 1st form by this invention. A state of superimposed display using a tablet in the first embodiment of the present invention. Explanatory drawing of the principle of a depth camera by the 2nd form by this invention. A state of superimposed display using a tablet equipped with a depth camera in the second embodiment of the present invention. (A) Data obtained by MRI converted to surface 3D polygon data, (b) Data taken by MRI converted to surface 3D point cloud data, (c) Taken by depth camera 3D point cloud data. Sectional drawing of a patient in endoscopic surgery. (A) Appearance of image when using laparoscope 13 (b) Appearance of image when using wide-angle laparoscope 12 (A) Explanatory drawing of the front-end | tip part of the wide angle laparoscope 12, (b) Explanatory drawing of the front-end | tip part of the laparoscope 13. FIG. Description of surgical instruments used for endoscopic surgery. Explanation when a marker group is attached to a surgical instrument used for endoscopic surgery. The state of the local coordinate measurement of each surgical instrument using the calibration stand 19 in the third mode of the present invention. A state of a cross-sectional view of a patient in an endoscopic operation, explaining an operation using the wide-angle laparoscope 12. (A) The appearance of the CCD when using the wide-angle laparoscope 12 and inserting the affected part measuring rod 12p, (b) the stored image when using the wide-angle laparoscope 12, and the tip position of the surgical instrument are shown. Explanatory drawing 1 of the 4th form by this invention. (A) Example of AR marker, (b) Marker diagram of square pattern with large black frame and explanatory diagram of measurement method. Explanatory drawing 1 of 1st Example of the 5th form by this invention. Explanatory drawing 2 of the 4th form by this invention. Explanatory drawing 2 of 1st Example of the 5th form by this invention. Schematic of the second embodiment of the fifth aspect according to the present invention. A description of a surgical instrument having an image input unit. A description of a surgical instrument having an image output unit. The perspective view when the Example of the 6th form by this invention is mounted | worn. The side view when the Example of the 6th form by this invention is mounted | worn. (A) Operation of the affected area when viewed with a binocular loupe, (b) 3D polygon data of the affected area taken by MRI, (c) The affected area when the embodiment of the sixth embodiment according to the present invention is mounted Superimposed display result. The figure when the shielding part is retracted in the embodiment of the sixth aspect according to the present invention. The figure at the time of inserting the shielding part in the Example of the 6th form by the present invention. (A) The front view of the Example of the 7th form by this invention, (b) The side view of the Example of the 7th form by this invention. (A) When the face is not moved in the front view of the embodiment of the seventh mode according to the present invention, (b) When the face is not moved in the side view of the embodiment of the seventh mode according to the present invention State. (A) When the face is moved in the front view of the embodiment of the seventh embodiment according to the present invention, the state when the stereoscopic microscope 47 is moving in an interlocked manner, (b) The embodiment of the seventh embodiment according to the present invention When the face is moved in the side view of FIG. Explanatory drawing of the control apparatus by this invention. (A) Explanatory drawing of the surgery system by a prior art, (b) The perspective view of a detachable binocular loupe, (c) The perspective view of the surgical stereomicroscope by a prior art. (A)-(c) Explanatory drawing of the procedure of the registration measurement by a prior art. (A) Example of display software for measurement image data by MRI according to the prior art, (b) State of confirmation work of registration measurement result according to the prior art. Surgery work to prevent infection.

  In the first embodiment, as shown in FIG. 1, a pole 2 is installed at a position that does not interfere with the operating room, and four motion capture sensor units 1 are attached. The sensor unit for motion capture 1 has infrared LED illumination. By illuminating the marker group 105 and the marker group 3 attached to the tablet 4 shown in FIG. Control is performed so that images of the three-dimensional marker group 105 and the marker group 3 can be captured.

  This motion capture control is performed by the motion capture control device 1PC. Images captured from the four motion capture sensor units 1 can be analyzed by the stereo image processing method, and the position coordinates of the marker group 105 and the marker group 3 can be accurately measured. It is sufficient if there are at least three sensor units 1 for motion capture, but the first and second surgeons and the supporting nurse are also around the patient 100 during the operation. Eight are arranged.

  Further, as shown in FIG. 2, the tablet 4 that is usually marketed has a built-in camera, and when the head of the patient 100 is taken with the built-in camera of the tablet 4, the head image of the patient 100 is displayed on the screen. The On the other hand, the MRI stores the internal video as image data with the human body cut into a circle. Therefore, the surface information of the human body can be converted into polygon data 9 by using the outer peripheral image as point cloud data. Similar to the prior art, by performing registration measurement, the head image of the patient 100 and the superimposed display image 5 based on MRI polygon data can be observed on the screen of the tablet 4.

  FIG. 4 shows an actual working scene when a depth camera 6 attached to the tablet 4 is used instead of the built-in camera. Time of Flight has been proposed as the simplest and best performance method for the depth camera 6. As shown in FIG. 3, this is a method of measuring the distance to the object by blinking the LED illumination 7 with a predetermined sin curve and detecting the phase delay D of the reflected light from the object. That is, in the case of the depth camera 6 having the two-dimensional light receiving element 8 with the objective lens, it is possible to detect the phase delay for each element, and the distance to the surface of the object is detected by the fineness of the number of divided elements. it can. Originally, the two-dimensional light receiving element 8 with the lens of the depth camera 6 itself has the position information of the top, bottom, left and right, and further, by taking the depth information by Time of Flight measurement, the surface information of the three-dimensional object 100 is obtained. Can be obtained.

  The second embodiment will be described. FIG. 6 is a cross-sectional view 16 of a patient in endoscopic surgery. Endoscopic surgery facilitates access to the affected area by opening several small holes of about 5mm to 2cm, and the role of the communication path that connects the body cavity and outside the body so that the injected carbon dioxide gas does not leak Is installed in the hole. From there, carbon dioxide gas is injected and inflated to secure the space necessary for the operation, and an endoscope and a thin surgical instrument are inserted to perform the operation. The endoscope is an instrument like a stomach camera, but the one dedicated to this operation is called a laparoscope 13 and projects the image of the affected part on a two-dimensional image element, thereby projecting the state of the affected part on the monitor. However, the conventional laparoscope 13 has a narrow field of view, and it takes time to align the laparoscope 13 and other thin surgical instruments to the affected area.

  As shown in FIG. 8 (a), the wide-angle laparoscope 12 according to the present invention employs a concave lens 12b in the light receiving portion at the tip, so that a wide visual field can be observed from the position of the hole as shown in FIG. Device. The tip portion shown in FIG. 8 (a) is similar to the tip portion 12a of the normal laparoscope 13 shown in FIG. 8 (b), which has a hole 13c and an illumination portion 13d used for instrument insertion and suction. 12a is provided with a hole 12c and an illumination unit 12d used for insertion and suction of an instrument, and by projecting an image of the affected part on a two-dimensional image element, the state of the affected part can be displayed on a monitor. The part which is not the convex lens 13b is different. Other CCD output units and the like have the same structure as the normal laparoscope 13.

  Further, as shown in FIG. 6, the thin portion to be inserted is shorter than the portion of the laparoscope 13 to the extent that the tip portion of the trocar 15 protrudes from the tip. Therefore, it can be seen that the structure is stable when inserted, and is suitable for projecting a wide field of view. FIG. 7B shows a state in which another surgical instrument is inserted while viewing the affected part with the wide visual field. Without getting lost, the tips of the laparoscope 13 and forceps 14 can be inserted to the affected area in a short time. In this state, when all the surgical instruments are inserted and aligned, the monitor image is switched to the image of the laparoscope 13 as shown in FIG. 7A, so that the alignment operation is shortened and smooth regardless of the organ position. It is possible to perform a surgical operation.

  Next, a third embodiment will be described. In the previous invention, when the wide-angle laparoscope 12 is used, a single trocar 15 is always occupied. However, the organ moves greatly because the patient is fixed in a predetermined shape and carbon dioxide gas is injected. Due to Therefore, if the wide-angle laparoscope 12 is first inserted and the position of the organ can be confirmed, as shown in FIG. 12, the wide-angle laparoscope 12 can be pulled out and the hole can be used for insertion of another surgical instrument. However, this makes it difficult to confirm the position when inserting each surgical instrument.

  Similarly to the calibrator 106 described above, if the tip portion a that performs a pointed tentacle is used as the tip (operation portion) of the surgical instrument, the marker group portion b is installed, and the interval is measured in advance as local coordinates, By checking the position coordinates of the marker group with the motion capture sensor unit 1, the world coordinate position of the distal end (action part) of the surgical instrument can be specified. However, these surgical instruments are always sterilized and used repeatedly. On the other hand, since the marker group has an irregular structure and the marker part is difficult to remove, when attached to a surgical instrument, the marker group is likely to be disposable or removed and cleaned and sterilized separately.

  FIG. 9 briefly introduces surgical instruments used for endoscopic surgery. Since the laparoscope 13 has been described above, the description thereof is omitted here. The wide-angle laparoscope 12 is provided with a hole 12c used for instrument insertion and suction, and a straight rod-shaped affected part measuring rod 12p can be inserted. The forceps 14 include a grasping forceps for grasping an object, a peeling forceps for peeling a tissue, a scissors-type forceps having a scissor function, and the like, and some of which have a function of an electric knife. The ultrasonic coagulation / cutting device 17 converts electrical energy into ultrasonic vibration and uses it for coagulation / cutting. Friction heat is generated by bringing the tip portion into contact with the tissue, and the tissue can be separated while coagulating (hemostasis). The electric knife 18 is an electric knife using a high frequency current. There are one electrode called monopolar and two electrodes called bipolar. Especially in the case of bipolar, it becomes possible to cauterize small lesions etc. by pinpoint, and heat damage other than the treatment part Risk is reduced.

  As described above, the marker groups 13e, 14e, 17e, and 18e in which the arrangement of the markers is slightly changed so that each surgical instrument can be identified as described above are appropriately used before surgery as shown in FIG. It will be attached. However, the marker group 12e is not attached to the wide-angle laparoscope 12 itself, but attached to the attached diseased part measuring rod 12p. Since the attachment position is attached via a double-sided tape at a position that does not interfere with the operation, local coordinate measurement of the marker group and the tip (operating portion) of the surgical instrument is not performed at the time of attachment.

  FIG. 11 shows a state of local coordinate measurement of each surgical instrument using the calibration stand 19 of the present invention. The calibration frame 19 includes a marker group 23 and an action portion installation point 20, and the local coordinate position is measured in advance. The action part installation point 20 is slightly depressed so that the tip of the action part can be stably installed. On the other hand, the support base 21 that supports the thin portion inserted into the trocar 15 has a V-shaped cut 22 that can be stably placed.

  As shown in FIG. 11, the standing surgical tool is supported by three points in total, that is, one point of the action portion installation point 20 and two circular contact points to the V-shaped cut portion, and can be stood stably. For the ultrasonic coagulation / cutting device 17 and the electric scalpel 18 having a heel shape at the tip, measurement is performed with the heel closed. At the stage of being stably placed, the world coordinate positions of the marker groups 12e, 13e, 14e, 17e, 18e and the marker group 23 are stored by the motion capture device. By subtracting the local coordinates of the marker group 23 and the action part installation point 20, the world coordinate position of the tip (action part) of each surgical instrument is obtained.

  As shown in FIG. 12, when the wide-angle laparoscope 12 is inserted, the whole image of the organ is confirmed, and where the affected part (black x in the figure) is located is confirmed. FIG. 13A shows the appearance of the image at that time. Since a circle mark 24 is displayed in advance at the center of the image, a wide-angle laparoscope is placed so that the affected area comes within the circle mark 24. The angle and position of 12 are moved, and the affected part measuring rod 12p is inserted into the hole 12c from the wide-angle laparoscope 12 until the tip reaches the affected part. After confirming that the tip has entered the circle mark 24, the world coordinate position and the image of the tip of the affected part measuring rod 12p are stored. That is, since the tip of the affected part measuring rod 12p is in contact with the affected part, the coordinate becomes the world coordinate position of the affected part.

  Although the stored image is shown in FIG. 13, it is an already stored image, and the stored image continues to be displayed on the monitor even when the wide-angle laparoscope 12 is pulled out from the trocar 15. Next, the laparoscope 13 is inserted. Since the world coordinate position of the distal end portion (light receiving portion) of the laparoscope 13 has already been obtained, the distal end coordinate position is superimposed and displayed in FIG. The position of the affected part and the tip part (light receiving part) of the laparoscope 13 is also displayed as position difference information, as in A (0, 0, +5). Similarly, for example, even if the forceps 14 or the electric knife 18 is inserted from another trocar 15, the distal ends are displayed with differently marked circles 25 and Δ 26, respectively, and the affected area and the distal ends of the forceps 14 and the electric knife 18 are displayed. The position of each part is also displayed as position difference information, such as B (+30, +40, +20) and C (−20, +20, +10). When these marks are almost in the circle, the image shown in FIG. 13 is switched to the image received by the laparoscope 13 as shown in FIG. 7 (a). It can be confirmed within the field of view.

  Next, a fourth embodiment will be described. In the stage of (4) explained in the prior art, as shown in FIG. 34, in order to prevent infectious diseases, other than the affected part is covered with a sterile cover 31, and further, the affected part is covered with a transparent sheet 32, so that it is fixed to the patient during the operation. It becomes impossible to see the marked marker 105. Since various treatments are performed during the operation, the position of the patient is often changed. However, according to the present invention, the second marker is attached to the transparent sheet 32, so that the second marker can be used even during the operation. The position of the patient can be confirmed, and control can be performed so that the superimposed image display is not shifted. For easy understanding in FIG. 14, the relationship between the size of the transparent sheet 32, the aseptic cover 31, the opening 31 a for the surgical operation, the affected part 29 (incision part), and the marker 30 with respect to the patient 100 and the three-point fixture 104. As you can see, everything is shown as a rectangle.

  First, before covering the patient 100 with the aseptic cover 31, registration is performed using the first and second embodiments. At this time, the superimposed display is performed without any problem. Next, the marker 30 is pasted in the vicinity of the affected part 29. Using the cameras used in the first to second embodiments, the position of the marker 30 is recognized as an image. Since the local coordinates of the marker group 105 and the MRI superimposed image 9 have already been obtained, if the local coordinates of the second marker 30 and the marker group 105 are obtained, the local coordinates of the marker 30 and the MRI image 9 are also obtained. That is, since the marker 30 is affixed to the head of the patient 100, the MRI image 9 and the head image of the patient 100 can be accurately superimposed and displayed by examining and correcting the local coordinate fluctuation amount of the marker 30. it can.

  As the second marker, here, the most known AR marker as shown in FIG. 15A is used. The AR marker refers to a pattern with some pattern drawn in a square black frame. It must be square. Basically, it has a black frame and the ratio of the white area inside is 1: 2: 1. Although the border can be changed, in general, the ratio of black frame to white area is 3: 14: 3, the lowest line is detected, the marker is detected with the black frame, and the marker is set according to the pattern in the white area. As a rule, it is determined that the pattern in the white region avoids point symmetry and line symmetry and does not use thin lines in the design portion.

  This is captured as an image and processed into an image to create a pattern file. A pattern file is a data file created from markers. For example, if an image in a white area has a resolution of 4 × 4, the whole is divided into 16 and digitized. Specify 16 for the basic numerical value. After that, the image is converted into a black and white binary image according to the threshold value, and the black and white image is inverted. Then, the white rectangular frame and the inside thereof are cut out, and the inner square is cut out from the square by the proportion of the black frame. This is a technique for comparing the cut rectangle and the value of the pattern file, calculating the coincidence rate, and performing image recognition. Increasing the resolution increases the recognition rate even for fine patterns, but also increases the load. If a plurality of AR markers are arranged, the measurement accuracy is improved due to the averaging effect.

  In FIG. 17, the sterile cover 31 is aligned so that the affected area 29 and the marker 30 are exposed from the opening 31a, and the other patient 100, the three-point fixture 104, and the marker group 105 are completely covered with the sterile cover 31. Further, the state where the affected part and the marker 30 are covered with the transparent sheet 32 is shown. Since the marker group 105 is hidden, the position of the head of the patient 100 cannot be detected by the motion capture sensor unit 1, but even if the position of the head of the patient 100 drifts, the camera 100 is used on the transparent sheet 32. By examining and correcting the fluctuation of the local coordinate value of the marker 30 that can be seen through, the MRI image 9 and the head image of the patient 100 can be accurately superimposed and displayed on the monitor.

  Although the AR marker is taken as an example this time, the AR marker is originally used for image recognition and the position accuracy itself is not so high. Originally, the amount of movement of the head of the patient 100 during the operation is not so large, and a large square pattern with a black frame as shown in FIG. 15B may be used instead of the AR marker. One-dimensional data X1, X2, Y1, and Y2 are formed by taking in this pattern as two-dimensional image data, dividing it like a dotted line, and performing vertical and horizontal addition processing for each divided region. By detecting the slope signal of the black frame portion and analyzing each slope position, it is possible to accurately detect not only the shift amount of x and y but also the rotation amount, pitching, and rolling error. Since the pattern is simple, it is not suitable for image recognition, but in a situation where the position of the pattern is fixed to some extent (a condition where the amount of head drift of the patient 100 is small), this pattern with high averaging effect can be obtained by performing addition processing. It is possible to maintain the superimposed display state with higher accuracy.

  Furthermore, in the fifth embodiment, since the second marker used in the fourth embodiment is formed on a plane, the measurement accuracy in the direction parallel to the plane is high, but the measurement accuracy in the direction orthogonal thereto is deteriorated. . The accuracy is higher when the marker group 34 originally formed in a three-dimensional manner is measured using the plurality of motion capture sensor units 1. Therefore, in the fifth embodiment, a connector portion 33 for attaching the marker group 34 is attached instead of the second marker, and the marker group 34 is attached to the connector portion 33 as necessary, and a plurality of motion capture sensor portions 1 are used even during the operation. Thus, the head position of the patient 100 can be accurately re-measured. The convex portion 33b of the connector portion 33 has a concave portion 34b made with high precision tolerance so that it can be attached and detached with good reproducibility only in a predetermined direction.

  In FIG. 16, before covering the patient 100 with the aseptic cover 31, registration is performed using the first and second forms. At this time, the superimposed display is performed without any problem. Next, the connector part 33 is affixed in the vicinity of the affected part 29, and the marker group 34 is attached. When the world coordinate positions of the marker group 105 and the marker group 34 attached to the connector unit 33 are measured using a plurality of motion capture sensor units 1, the local coordinate values of the marker group 105 and the MRI superimposed image 9 have already been obtained. The local coordinate values of the marker group 34 and the MRI superimposed image 9 are obtained.

  The marker group 34 is removed from the connector part 33, covered with a sterile cover 31, and the transparent sheet 32 is pasted from above the connector part 33. The connector part 33 has a convex part 33b for attaching the marker group 34 and a wide pasting area. There is a flat surface portion 33a. When the transparent sheet 32 is pasted from above the connector portion 33, a convex portion 33b hollow portion as shown by a dotted line is generated in the flat portion 33a. When the transparent sheet 32 is cut along the dotted line, the convex portion 33b is exposed. However, since the flat part 33a and the adhesive back surface of the transparent sheet 32 are firmly adhered to the dotted line 32a, there is no problem that the bacteria on the back side are exposed to the surface during the operation.

  In FIG. 18, the aseptic cover 31 is aligned so that the affected part 29 and the connector part 33 are exposed from the opening 31a, and the other patient 100, the three-point fixture 104, and the marker group 105 are completely covered with the aseptic cover 31. In addition, after the affected part and the marker 30 are further covered with the transparent sheet 32, it is cut along a dotted line 32a including the convex part 33b portion of the connector part 33 to expose the convex part 33b. Since the marker group 105 is hidden, the head position of the patient 100 cannot be detected by the motion capture sensor unit 1, but the marker group 34 is attached to the convex portion 33b even if the position of the head of the patient 100 drifts. The MRI image 9 and the head image of the patient 100 can be accurately superimposed and displayed on the monitor by examining and correcting the fluctuation amount of the marker group 34 using the plurality of motion capture sensor units 1.

  In FIG. 19, the fifth mode is realized by another method. The vicinity of the affected area 29 is an important area at the time of surgery, and it may be difficult to attach the marker 30 and the connector section 33 there. Therefore, in FIG. 19, the head away from the affected area 29 is incised, a screw hole is dug in the skull, and the connector part 35 having a slightly higher strength is embedded. Since the entire patient 100 is covered with the aseptic cover 31, an arm part 36 that can be connected to the connector part 35 is inserted, and relayed to a position outside the aseptic cover and without interfering with the operation. A marker group 37 is attached to the tip as a connector portion. The installation itself is a process that takes almost no time. During installation, the arm is supported by a person so that no load is generated on the skull part where the screw hole is dug.

  Next, as a superimposition display method, an image input unit and an image output unit are required in order to view an image superimposed and displayed during the operation. FIG. 20 shows a surgical instrument having an image input unit, which includes a CCD camera of the laparoscope 13, a built-in camera of the tablet 4, a video camera 110 for surgical recording, a recording camera for the surgical microscope 38, and the like. On the other hand, FIG. 21 shows a surgical instrument having an image output unit, which includes a surgical head mounted display 39, a tablet 4, a PC monitor 111, and the like.

  Using the image composition device, the 3D image obtained from MRI and the image data obtained from the image input unit are superimposed and displayed according to the result of the motion capture device. FIG. 22 shows the sixth embodiment. FIG. 23 is a side view of an embodiment of the sixth mode. This is a structure in which a Kepler type (magnification 4 to 6) magnifying part (also a Galileo binocular loupe type) 42 and an objective lens part 44 are joined by an image synthesizing part 45. The in-focus point of the loupe unit, the two-dimensional image display liquid crystal unit 43 attached to the tip of the objective lens 44 and the retina of the eye are in a conjugate position, and are configured as a pair for the right eye and the left eye. A configuration in which these optical portions are held by the detachable portion 41, and the entire image output device 40 can be stably positioned at the positions of both eyes by connecting the detachable portion 41 to the spectacle-type fixing portion 112 that can be attached. It has become. In addition, a pair of built-in cameras 63 are also installed in the detachable section 41, and information obtained there can be displayed on the two-dimensional image display liquid crystal section 43.

  Further, the weight of the loupe unit and the glasses-type fixing unit 112 alone is 70 to 90 g, and when the objective lens unit 44 is combined with the image synthesizing unit 45 and the two-dimensional image display liquid crystal unit 43, the weight is about 200 g. It is difficult to support this beside the ears and nose as with glasses. Therefore, in this image output device 40, an elastic rod 60 made of a shape memory alloy is attached to the detachable portion 41, and the fulcrum portion 59 protruding from the support surface 58 on the head of the head is used as a fulcrum. The balance between the weight and the balancer 61 is maintained in the form of “Yajirobe”. The position of the elastic bar 60 and the fulcrum, and the position and weight of the balancer 61 can be finely adjusted. It becomes possible to keep wearing it stably.

With this structure, all the weight is applied to the support surface 58 under the fulcrum 59, but even if the weight of the balancer is combined, it is as heavy as a construction helmet and the area of the support surface 58 is large. The whole head will support the weight. There is no discomfort due to imbalance or tightening like a binocular loupe or head-mounted display, and it can be worn for a long time even during surgery.
Further, as shown in FIG. 23, like a normal binocular loupe, by folding the connector part, the loupe part 42 can be retracted from the line of sight of the eye as in the binocular loupe, and a wide field of view (naked eye or glasses) is obtained. ) To continue the surgery.

  First, the image output device 40 is attached to the glasses-type fixing unit 112, and is worn in front of the face with one hand as if wearing glasses. Next, since the surgical cap is usually worn, the support surface 58 is pulled up with the other hand, and the adhesive portion attached to the back side of the support surface 58 is attached to the surgical cap at a position where the weight feeling of the image output device 40 is almost eliminated. Attach to the pressing surgical cap. Then, by finely adjusting the position of the fulcrum bar 59 and the position of the balancer 61, it is only necessary to confirm a state that fits in both the extended state and the bent state.

  Since the marker group 34 is also attached to the attachment / detachment portion 41 of the image output device 40, registration can be performed. If the image from the built-in camera 63 and the 3D image obtained from the MRI are displayed as a superimposed image on the two-dimensional image display liquid crystal unit 43 by registration, all the conditions necessary for carrying out the first and fifth embodiments are obtained. organized. Furthermore, if the built-in camera 63 is changed to the depth camera 6, the second embodiment can be implemented. By allowing the built-in camera 63 to detect the AR marker, the fourth embodiment can be implemented.

  Furthermore, if the video from the CCD camera of the laparoscope 13 and the tip information of the surgical instrument are displayed on the two-dimensional image display liquid crystal unit 43 as a superimposed image, the third embodiment can be implemented. However, in this case, the appearance of the loupe unit 42 is obstructive. As shown in FIGS. 25 and 26, the image output device 40 is provided with a shield plate 46 that can be inserted and removed between a composite mirror (half mirror) 45 and a loupe unit 44. When it is desired to view only the image from the CCD camera of the laparoscope 13 as in the case of endoscopic surgery, a shielding plate 46 is inserted as shown in FIG. The split mirror may be a polarizing mirror, and a transmissive liquid crystal element may be placed in the 46 portion. Since the transmitted light of the polarizing mirror is P-polarized light and the image coming from the liquid crystal unit 43 for two-dimensional image display is S-polarized light, the same can be done if the P-polarized light is electrically shielded by the transmissive liquid crystal element unit. On the other hand, when only the appearance of the loupe unit 42 is desired to be seen, the image of the two-dimensional image display liquid crystal unit 43 is turned off and the light shielding plate 46 is retracted as shown in FIG. In the case of using a transmissive liquid crystal element portion, it is sufficient to electrically transmit P-polarized light.

  In addition, since the image output device 40 can observe the actual intraoperative state from the loupe unit 42, the image of the liquid crystal unit 43 for two-dimensional image display can be superimposed on the appearance. For this usage, a separate registration work is required. Since the interval between the pair of loupes 42 and the interval between the built-in cameras 63 arranged in a pair are substantially equal, when these are superimposed and displayed, both images can be seen by the half mirror. When the view center of the loupe unit 42 is viewed, it is designed to be parallel to the view center of the built-in camera, and the position and magnification of the image on the liquid crystal unit 43 for two-dimensional image display are matched so that its appearance and magnification match. Adjust. If the appearances match, the registration work ends.

  The last thing to do is when using a surgical microscope for two-person observation in advanced surgery. If the first surgeon and the second surgeon can see the three-dimensional image of the affected area at the same time, and can see the superimposed display image at the same time, efficient and safe navigation is possible.

  27 (a) to 27 (b) are examples in the seventh embodiment, (a) is a front view, and (b) is a side view. When the light of the LED illumination 62 irradiates the observation target T, the diffuse reflected light from the target T becomes a parallel light flux through the deflecting lens 48 as a light flux having a predetermined inclination. Thereafter, the image of the target T is projected onto the two-dimensional CCD 51 through the zoom optical system 49 and the objective / focus adjustment lens 50. This is because the distance between the light beams in the deflection lens 48 is set to be 62 mm, which is the distance between the eyes of the person, and the stereoscopic vision when viewing the target T with both eyes positioned at the deflection lens 48 is almost the same. is there.

  That is, when the image of the two-dimensional CCD 51 is viewed on, for example, the head mounted display type image display device 40 described in the sixth embodiment, the operation is performed while looking at the target T with both eyes placed at the position of the deflection lens 48. The conditions are almost equivalent to. Since the image of the two-dimensional CCD 51 can be electrically distributed, all persons wearing a head mounted display type image display device can view a stereoscopic image under the same conditions.

  Further, in this stereoscopic microscope 47, when the left and right sides of the paper surface are the X axis, the top and bottom are the Y axis, and the depth is the Z axis, an LM block 53b1 pair is installed, and the LM rail 53a on the rotation stage 53c around the Z axis Above, the stereo microscope 47 is supported so as to be rotatable around the X axis with the target T as the rotation center, thereby forming an R guide. In addition, an LM block 52b1 pair is installed on the Z-axis rotation stage 53c, and can rotate about the Z-axis around the target T on the LM rail 52a on the X-axis movement stage 54X. To form an R guide. With these two guides, the stereoscopic microscope 47 can rotate around the X axis and the Z axis with the target T as the rotation center.

  Further, a linear guide is disposed on the X-axis moving stage 54X and supported by a linear rail on the base portion 54. The base portion 54 is supported by an extension rod 55 for separating the work area and the support portion of the stereoscopic microscope 47 so as not to obstruct the surgeon. A Z-direction moving shaft 56 is fixed to the extension rod 55. The support shaft 57 can be moved in the Z-axis direction by a linear guide.

  A driving device and an encoder are attached to each guide part, and the stereoscopic microscope 47 is moved in the X-axis direction and the Z-axis direction by rotating around any X axis and Z axis around the target T as the rotation center. It is a possible configuration. The support shaft 57 is supported by a movable stereomicroscope support unit (not shown) installed outside the work area, but can be driven in each direction for installing the stereomicroscope on the operating table. A Z-direction drive unit, a Y-axis rotation drive unit, wheels, and the like are mounted.

  Since the image display device 40 is a head-mounted display type, an image can be viewed at an arbitrary position. However, when the surgeon performs an operation, the affected part of the patient 100 is at the position of the target T, and the fingertip needs to work at that position. Therefore, the positional relationship between the stereoscopic microscope 47 and the image display device 40 is as shown in FIGS. (A) is a front view, and (b) is a side view. Here, if the distance between the deflection lens 48 and the target T is L, the surgeon will say that the virtual target is TT that is L away from the line of sight in the most comfortable line-of-sight state (usually 20-30 degrees downward). Become. The direction of the eyes and TT is the Z-axis direction in the image display device 40, and the angles of the X-axis and the Y-axis are determined accordingly, and this is the coordinate axis B in the virtual space.

  Since the surgeon sees the affected part of the patient 100 at the position of TT on the coordinate axis, the surgeon can control the stereoscopic microscope 47 in the same direction as the direction of the face during the operation. Since the affected part can be seen according to the line of sight, not only the efficiency and safety of the work are improved, but also the effect of preventing virtual sickness. FIGS. 29A and 29B show how to move a normal surgeon's face. The horizontal movement around the Z axis with reference to the coordinate axis B with respect to the virtual target TT is a rotation along the paper surface of FIG. 29A and a lateral shift along the paper surface of the stereoscopic microscope 47, and therefore a rotation stage around the Z axis. The rotation around the Z axis of 53c and the X drive of the X axis moving stage 54X can be handled only.

  Further, the vertical movement around the X axis with reference to the coordinate axis B with respect to the virtual target TT is a rotation along the paper surface of FIG. 29B and a vertical shift along the paper surface of the stereoscopic microscope 47, and thus the body microscope 47. The rotation around the X axis and the Z drive of the Z-direction moving shaft 56 can be handled. Since each drive system can be controlled independently, smooth drive control of the stereoscopic microscope 47 can be performed according to the movement of the face. For the operation of moving the face closer to or away from the virtual target, the magnification itself is varied by controlling the zoom optical system.

  For the movement of the face, the movement of the marker group 34 installed in the image display device 40 is measured using a plurality of motion capture sensor units 1, and the stereoscopic microscope 47 is driven and controlled according to the fluctuation position and fluctuation angle with respect to the coordinate axis B reference. Just do it. It is also possible to feedback control whether or not the stereoscopic microscope 47 is driven as expected by installing a marker group in the stereoscopic microscope 47 and measuring it using a plurality of motion capture sensor units 1. Although the image display device 40 is used this time, it goes without saying that the same thing can be achieved by using a normal surgical head mounted display 39.

  As for the superimposed display when viewing the image of the stereoscopic microscope 47 with the image display device 40, if a marker group is installed in the stereoscopic microscope 47, the two-dimensional CCD 51 originally functions as a built-in camera. Can be measured using the third to fifth aspects of the invention. However, the zoom optical system 49 is mounted on the stereoscopic microscope 47, and when the magnification of the image is changed by the zoom optical system 49, the superimposed display control using the marker group cannot be performed. In this case, by confirming the AR marker used in the fourth form arranged in the vicinity of the target T with the two-dimensional CCD 51 and correcting the magnification and the position change, stable superimposed display is possible.

  In addition, since the drive unit of the stereo microscope 47 can be controlled independently without interfering with each direction, it automatically measures in advance how much the superimposed display image drifts according to the variation of each encoder value. The variation error may be stored as a map. Regarding errors due to mechanical tolerances and deformation of the self-weight of the stereoscopic microscope 47, the drift of the superimposed display image can be minimized by correcting with a variation error map by automatic measurement in that state.

  Next, a control method for realizing the first to seventh embodiments according to the present invention will be described with reference to FIG. Image data of the patient 100 taken by the MRI 113 is converted and stored by the MRI image converter 64 into polygon data that can be displayed in 3D and surface point cloud data for matching. On the other hand, the shape of the marker group attached to the surgical instrument, the calibration stand 19, the image output device 40, etc. is detected by the motion capture sensor unit 1 according to a command from the motion capture control device 1PC. It is registered in the storage device of 1PC.

  The image display device 40 has a function of recording data taken by the built-in camera 49 in the image input / output control device 40PC in response to a command from the image input / output control device 40PC, and data is recorded in the image input / output control device 40PC. The image data is displayed from the two-dimensional image display liquid crystal unit 43. The operation stereo microscope 47 has a function of recording data taken by the two-dimensional CCD 51 in the stereo microscope control apparatus 47PC in accordance with a command from the stereo microscope control apparatus 47PC, and a position where the attitude of the operation stereo microscope 47 itself is determined. It has a position control function to drive.

  The motion capture control device 1PC creates a predetermined world coordinate system in the measurable region, measures the world coordinate position of each marker group in the marker by the motion capture sensor unit 1, and sends the result to the system control device MPC. Always output. The system control device MPC has an input device for inputting in advance which marker group is used and what the local coordinates of the calibration base 19 are. When the marker group to be used is placed on the calibration base 19, the world coordinates of the marker group are automatically measured, and the world coordinate position when the value is stabilized is stored. By subtracting the local coordinates of the calibration base 19, it has a function of detecting the world coordinate position of the distal end portion (action portion) of each surgical instrument.

  In addition, the system control device MPC takes in the image information stored in the image display control device 40PC, the stereoscopic microscope control device 47PC, and the MRI image conversion device 64 and performs matching to thereby measure the displacement amount of different coordinate axes. It has a function. Further, according to the result, the polygon data stored in the MRI image conversion device 64 is shifted, rotated and corrected for magnification, and the image information obtained from the image display control device 40PC and the stereomicroscope control device 47PC is obtained. It has a function of synthesizing images by performing transmission processing and outputting the synthesized data to the image display control device 40PC and the stereomicroscope control device 47PC again.

  Further, regarding the position control of the stereoscopic microscope apparatus 47 using the image display control apparatus 40, the position of the marker group installed in the image output apparatus 40 is detected, and the stereoscopic microscope control apparatus 47PC determines how much three-dimensionality according to the result. It can be determined whether to move the microscope apparatus 47. For the control, there is a pedal at the foot of the surgeon that instructs on / off of position control. When it is on, it is driven according to the movement of the face, but when it is off, it is not controlled by the movement of the face. Conditions can also be set. In addition, when the work of tilting the face is cramped, the setting of each other's coordinate axes can be changed with the reset button.

1 ... motion capture sensor,
1PC: Motion capture control device,
2 ... Paul,
3, 12e, 13e, 14e, 17e, 18e, 23, 34, 37, 105 ... marker group,
4 ... Tablet,
5 ... superimposed display image,
6 ... depth camera,
8: Two-dimensional light receiving element with objective lens,
9 ... Polygon data,
12 ... Wide-angle laparoscope,
12p ... affected area measuring rod,
13 ... Laparoscope,
14 ... forceps,
15 ... Troker,
17 ... Ultrasonic coagulation / cutting device,
18 ... Electric knife,
19: Calibration stand,
20 ... Action part installation point,
22 ... V-shaped notch,
29 ... affected area,
30 ... Marker
31 ... Aseptic cover,
32 ... Transparent sheet,
33, 35 ... connector part,
36 ... arm part,
38 ... surgical microscope,
39: Surgical head-mounted display,
40: Image output device,
40PC: Image input / output control device,
41 ... Desorption part,
42 ... Loupe part,
43... 2D image display liquid crystal unit,
44 ... objective lens part,
45... Image compositing unit,
46: shielding plate,
47 ... Stereo microscope,
47PC ... Stereoscopic microscope control unit,
48 ... deflection lens,
49 ... zoom optical system,
50: Objective / focus adjustment lens,
51 ... 2D CCD,
54 ... Base part,
54X ... X-axis moving stage,
58 ... support surface,
59 ... fulcrum part,
60 ... elastic bar,
61 ... Balancer,
62 ... LED lighting,
63 ... Built-in camera,
64 ... MRI image conversion device,
100 ... Patient,
104 ... 3-point fixture,
106: Calibrator,
107 ... 3D measuring device,
109 ... Operating table,
110 ... Surgery recording video camera,
111 ... PC monitor,
112 ... glasses-type fixing part,
113 ... MRI
MPC ・ ・ ・ System controller

Claims (7)

A motion capture device having an imaging unit that emits infrared rays to a predetermined position in the operating room and receives reflected light from a marker group arranged at a predetermined interval is disposed in at least three locations,
An image display device that superimposes and displays a predetermined first image on the affected area of the patient undergoing surgery, and the patient's affected area and the superimposed display image according to the coordinate position and rotation amount of the marker group measured by the motion capture device In a surgical system having an image display device that changes
The image display device includes a camera unit that captures an image of the affected area, a depth camera unit that measures the depth of the affected area, and the first marker group. A motion capture device having an imaging unit that emits infrared rays to a predetermined position in the operating room and receives reflected light from marker groups arranged at predetermined intervals is arranged, and a predetermined first is applied to an affected part of a patient undergoing surgery. Surgical system having an image display device that superimposes and displays an image, and having an image display device that changes the affected area of the patient and the superimposed display image according to the coordinate position and rotation amount of the marker group measured by the motion capture device In
A first endoscope having a distal end made of a concave lens having a first marker group, a step of identifying an affected area from an image of an entire organ taken by the first endoscope; and The second endoscope or surgical instrument for displaying an enlarged image has a second marker group, the insertion direction of the second endoscope or surgical instrument or the position of the distal end portion and the identified affected part. An endoscope apparatus for a surgical operation system, characterized by displaying a state of a relative position with respect to the position of the surgical system. At least three motion capture devices having imaging units that emit infrared rays at predetermined positions in the operating room and receive reflected light from marker groups arranged at predetermined intervals are arranged on the affected area of a patient undergoing surgery. An image display device that includes an image display device that superimposes and displays a predetermined first image, and that changes an affected display area of a patient and a superimposed display image according to a coordinate position and a rotation amount of the marker group measured by the motion capture device. A surgical system comprising:
A medical device for performing an operation on an affected area of a patient has a first marker group that can be detached,
The calibration device that can be supported by the action part of the medical device and at least two other points has a second marker group, and a portion that contacts the action part of the medical device and a relative coordinate position of the second marker group. A surgical operation characterized by storing in advance, and measuring and storing the position coordinates of the first marker group and the second marker group when the medical device is placed on the calibration device with the motion capture device. Method. A motion capture device having an imaging unit that emits infrared rays to a predetermined position in the operating room and receives reflected light from marker groups arranged at predetermined intervals is arranged, and a predetermined first is applied to an affected part of a patient undergoing surgery. Surgical system having an image display device that superimposes and displays an image, and having an image display device that changes the affected area of the patient and the superimposed display image according to the coordinate position and rotation amount of the marker group measured by the motion capture device In
The image display device includes a camera that captures an image of the affected area and the first marker group, and covers the affected area of the patient in order to superimpose and display the second image captured by the camera and the first image. A surgical operation method comprising attaching at least one or more second markers to a transparent sheet pasting region. At least three motion capture devices having imaging units that emit infrared rays at predetermined positions in the operating room and receive reflected light from marker groups arranged at predetermined intervals are arranged on the affected area of a patient undergoing surgery. An image display device that includes an image display device that superimposes and displays a predetermined first image, and that changes an affected display area of a patient and a superimposed display image according to a coordinate position and a rotation amount of the marker group measured by the motion capture device. A surgical system comprising:
The image display device includes a camera that captures an image of the affected area and the first marker group. In order to superimpose and display the second image captured by the camera and the first image, from the affected area of the patient A marker system connector device for a surgical operation system, wherein a connector portion capable of fixing a second marker group is attached to a part of a patient who does not go through a joint portion. At least three motion capture devices having imaging units that emit infrared rays at predetermined positions in the operating room and receive reflected light from marker groups arranged at predetermined intervals are arranged on the affected area of a patient undergoing surgery. An image having an image display device that superimposes and displays a predetermined first image, and an image that changes the affected area of the patient and the superimposed display image according to the coordinate position of the marker group and the rotation amount of the mark group measured by the motion capture device In a surgical system having a display device,
The image display device includes first and second magnifying optical systems for independently viewing the affected area with the left and right eyes, and between the first and second magnifying optical systems and the eyes, An image display device for a surgical operation system, comprising first and second composite mirrors for displaying a projection image and the first image in a superimposed manner, and further comprising a first marker group. At least three motion capture devices having imaging units that emit infrared rays at predetermined positions in the operating room and receive reflected light from marker groups arranged at predetermined intervals are measured by the motion capture device. In the surgical operation system having an image display device that changes the display image of the affected area of the patient according to the coordinate position of the marker group and the rotation amount of the mark group,
The image display device is a head mounted display in which a first marker group is arranged,
A stereoscopic microscope for imaging an affected area of a patient has a two-dimensional image sensor that receives an image of the affected area on the first imaging plane, and the affected area is detected according to the movement of the face of the head-mounted display wearer. A surgical operation characterized by having a control mechanism for controlling the rotation and shift amount of the stereomicroscope with the rotation center as the center, and controlling the rotation and shift amount of the stereomicroscope according to the movement of the face of the wearer apparatus.