JP6548110B2 - Medical observation support system and 3D model of organ - Google Patents

Description

  The present invention relates to a medical observation support system for supporting observation of an organ in a subject and a three-dimensional model of the organ, and in particular to an improvement for realizing highly practical observation support by intuitive operation.

  With the development of medical imaging devices in recent years, medical images such as three-dimensional virtual images have come to be widely used in the fields of diagnosis and surgery. In such a technique, a computer based on computer graphic technology based on image data of an organ captured in advance by a computed tomography (CT) device or a magnetic resonance imaging (MRI) device or the like, or such image data The anatomical structure of the organ in the subject can be grasped from the three-dimensional virtual image or the like reproduced on the screen of.

  In recent years, three-dimensional molding apparatuses that create three-dimensional objects (three-dimensional objects) based on three-dimensional data have rapidly developed and are attracting attention. An example is called a three-dimensional rapid prototyping machine or a three-dimensional printer (3D printer). In particular, according to the three-dimensional shaping apparatus capable of producing the three-dimensional object using a transparent material, for example, with respect to a model of an organ, a lesion site such as a blood vessel or a tumor inside the organ is colored material and surrounding tissue is transparent material By configuring each of them, it is possible to create a three-dimensional model in which blood vessels, tumors and the like inside the organ can be viewed from the outside. According to such a technique, it is possible to create a three-dimensional model as a so-called made-to-order model that reflects the characteristics of the organ of each subject, and can be used to support diagnosis and surgery (see, for example, Non-Patent Document 1).

Igami T, Nakamura Y, et al. Application of a Three-dimensional Print of a Liver for Hepatectomy for Small Tumors Invisible by Intraoperative Ultrasonography: Clinical Experience. ′ ′ World Journal of Surgery, _DOI. , 2014

  In the medical image display technology, for example, in order to view a three-dimensional image from a desired direction, the three-dimensional image is moved by designating rotation with respect to a plurality of axes preset in the three-dimensional image space or Or, a relatively complicated procedure is required, such as redrawing by specifying the viewpoint position in the three-dimensional image space. In the three-dimensional virtual image, processing such as quantitative measurement or display of a cut surface is easily realized. For example, at the time of surgery, it is necessary to measure the length of a blood vessel and the like in the vicinity of a region to be ablated, but in the three-dimensional virtual image, by specifying a segment to be a target on the computer screen, Calculation of the length of the section and display of the calculation result are performed. However, such designation also requires, for example, the position in the three-dimensional image to be performed using an input device such as a mouse, resulting in a relatively complicated procedure or in the two-dimensional plane on which the three-dimensional image is displayed. In a display, it may be difficult to input a desired position accurately.

  The three-dimensional model of the organ created by the three-dimensional molding apparatus is useful for grasping the shape of the organ and its three-dimensional effect. However, in such a three-dimensional model of an organ, it is difficult to perform operations such as quantitative measurement or display of a cut surface. For example, in the case of measuring the length of a blood vessel or the like in the vicinity of a region to be ablated during surgery, a method of measuring by applying a wire for measuring the length to a three-dimensional model of the organ may be employed. Task is not practical. On the other hand, in the case of using a three-dimensional image, it is considered relatively easy to calculate the distance on the three-dimensional image, but the input as to which distance between points is to be measured is difficult as in the above case.

  The present invention has been made against the background described above, and the object of the present invention is to provide a medical observation support system and a three-dimensional model of an organ for realizing highly practical observation support by intuitive operation. It is to do.

In order to achieve such an object, the first aspect of the present invention is a medical observation support system for supporting observation of an organ in a subject, which is based on three-dimensional image data of the organ obtained in advance. A position determination unit that determines a position on the model indicated by a predetermined instruction unit in the three-dimensional model of the organ created by the three-dimensional molding apparatus and the three-dimensional model of the organ; three-dimensional image data of the organ Based on the display unit for generating and displaying the image of the organ, and correlating control for correlating the position on the model with the position in the image of the organ displayed by the display unit or related information related to the position And a distance calculating unit that calculates a distance corresponding to the position on the model in the image of the organ .

According to the first aspect of the invention, predetermined instructions are provided in the three-dimensional model of the organ created by the three-dimensional molding apparatus based on the three-dimensional image data of the organ obtained in advance, and the three-dimensional model of the organ A position determination unit that determines a position on the model indicated by the unit, a display unit that generates and displays an image of the organ based on three-dimensional image data of the organ, and the display unit displays the position on the model And the association control unit for correlating with the position in the image of the organ to be detected or the related information related to the position, the instruction unit using the three-dimensional model of the organ as an input interface The operation indicating the three-dimensional model of the organ can be reflected on the image of the organ displayed by the display unit. That is, it is possible to provide a medical observation support system which realizes highly practical observation support by intuitive operation. Further, according to the first aspect of the invention, a distance calculation unit is provided which calculates a distance corresponding to the position on the model in the image of the organ. In this way, using the three-dimensional model of the organ as an input interface, the distance corresponding to the operation of the instruction unit can be easily calculated.

In addition, in order to achieve the above object, the subject matter of the second invention is a medical observation support system for supporting observation of an organ in a subject, wherein three-dimensional image data of the organ obtained in advance is used. Based on the three-dimensional model of the organ created by the three-dimensional molding apparatus and the three-dimensional image of the organ, a position determination unit that determines the position on the model indicated by the predetermined instruction unit in the three-dimensional model of the organ A display unit that generates and displays an image of the organ based on data, and a correlation that associates the position on the model with a position in the image of the organ displayed by the display unit or related information related to the position A control unit, the display unit generating and displaying a three-dimensional virtual image of the organ based on the three-dimensional image data of the organ; Flip and the incision line, is intended to be synthesized and displayed in the three-dimensional virtual image of the organ. According to the second aspect of the present invention, predetermined instructions are provided in the three-dimensional model of the organ created by the three-dimensional shaping apparatus and the three-dimensional model of the organ based on the three-dimensional image data of the organ obtained in advance. A position determination unit that determines a position on the model indicated by the unit, a display unit that generates and displays an image of the organ based on three-dimensional image data of the organ, and the display unit displays the position on the model And the association control unit for correlating with the position in the image of the organ to be detected or the related information related to the position, the instruction unit using the three-dimensional model of the organ as an input interface The operation indicating the three-dimensional model of the organ can be reflected on the image of the organ displayed by the display unit. That is, it is possible to provide a medical observation support system which realizes highly practical observation support by intuitive operation. Further, according to the second invention, the display unit generates and displays a three-dimensional virtual image of the organ based on three-dimensional image data of the organ, and the transition of the position on the model is performed. The corresponding incision line is displayed synthetically on the three-dimensional virtual image of the organ. In this way, using the three-dimensional model of the organ as an input interface, an incision line corresponding to the operation of the pointing unit can be simply displayed on the three-dimensional virtual image of the organ.

The first invention or where the gist of the third invention is dependent on the second shot bright, the display unit, based on a 3-dimensional image data of the organ, and generating a three-dimensional virtual image of the organ It is displayed, and the position on the model is synthesized and displayed on a three-dimensional virtual image of the organ. In this way, it is possible to simply display the position indicated by the pointing unit on the three-dimensional virtual image of the organ using the three-dimensional model of the organ as an input interface.

According to a fourth aspect of the present invention according to any one of the first to third aspects, the display unit generates a tomographic image of the organ based on three-dimensional image data of the organ. To display the position on the model on the tomographic image of the organ. In this way, the three-dimensional model of the organ can be used as an input interface, and the position indicated by the pointing unit can be simply synthesized and displayed on the tomographic image of the organ.

The subject matter of the fifth invention according to any one of the first to fourth inventions comprises an attitude sensor for detecting an attitude of the three-dimensional model of the organ with reference to a predetermined direction, The display unit generates and displays a three-dimensional virtual image of the organ based on three-dimensional image data of the organ, and a three-dimensional virtual image of the organ based on a detection result by the posture sensor. Change the attitude of By so doing, display control can be realized in which the posture of the three-dimensional virtual image of the organ follows the posture of the three-dimensional model of the organ, using the three-dimensional model of the organ as an input interface.

The subject matter of the sixth invention according to the fifth invention is that a sterile bag member is provided, and the three-dimensional model of the organ and the posture sensor are enclosed in the bag member. In this way, the three-dimensional model of the organ can be used as an input interface even during surgery.

The gist of the seventh invention according to any one of the first to sixth inventions is a plurality of markers attached to the three-dimensional model of the organ, and a marker attached to the pointing portion And a marker detection device that detects the position of the marker, and the position determination unit determines the position on the model based on the detection result by the marker detection device. In this way, it is possible to determine the position indicated by the pointing unit in the three-dimensional model of the organ in a practical manner.

The gist of the eighth invention according to the seventh invention is that the marker is a magnetic marker, and the marker detection device is a device for magnetically detecting the marker. In this way, it is possible to determine the position indicated by the pointing unit in the three-dimensional model of the organ in a practical manner.

The gist of the ninth invention according to the seventh invention is that the marker is an optical marker, and the marker detection apparatus is an apparatus for optically detecting the marker. In this way, it is possible to determine the position indicated by the pointing unit in the three-dimensional model of the organ in a practical manner.

Further, in order to achieve the above object, the present invention is a medical observation support system for supporting observation of an organ in a subject, which is a three-dimensional image data of the organ obtained in advance. Based on the three-dimensional model of the organ created by the three-dimensional molding apparatus and the three-dimensional image of the organ, a position determination unit that determines the position on the model indicated by the predetermined instruction unit in the three-dimensional model of the organ A display unit that generates and displays an image of the organ based on data, and a correlation that associates the position on the model with a position in the image of the organ displayed by the display unit or related information related to the position A control unit, a plurality of markers attached to the three-dimensional model of the organ, a marker attached to the instruction unit, and a marker detection device for detecting the position of the marker The position determination unit determines the position on the model based on the detection result by the marker detection device, the marker is a two-dimensional code, and the marker detection device includes an imaging device and a two-dimensional code. It is a code reader. According to the tenth aspect, based on the three-dimensional image data of the organ obtained in advance, the three-dimensional model of the organ created by the three-dimensional molding apparatus and the three-dimensional model of the organ A position determination unit that determines a position on the model indicated by the unit, a display unit that generates and displays an image of the organ based on three-dimensional image data of the organ, and the display unit displays the position on the model And the association control unit for correlating with the position in the image of the organ to be detected or the related information related to the position, the instruction unit using the three-dimensional model of the organ as an input interface The operation indicating the three-dimensional model of the organ can be reflected on the image of the organ displayed by the display unit. That is, it is possible to provide a medical observation support system which realizes highly practical observation support by intuitive operation. Further, according to the tenth aspect of the present invention, it comprises: a plurality of markers attached to the three-dimensional model of the organ; a marker attached to the instruction unit; and a marker detection device for detecting the position of the marker The position determination unit determines the position on the model based on the detection result by the marker detection device. In this way, it is possible to determine the position indicated by the pointing unit in the three-dimensional model of the organ in a practical manner. Further, according to the tenth aspect of the present invention, the marker is a two-dimensional code, and the marker detection device is an imaging device and a two-dimensional code reader. In this way, it is possible to determine the position indicated by the pointing unit in the three-dimensional model of the organ in a practical manner.

In order to achieve the above object, according to the eleventh aspect of the present invention, there is provided a three-dimensional model of the organ created by the three-dimensional molding apparatus based on three-dimensional image data of the organ obtained in advance. A position determination unit that determines a position on the model indicated by a predetermined instruction unit in the three-dimensional model of the organ, and a display unit that generates and displays an image of the organ based on three-dimensional image data of the organ A control unit for correlating the position on the model with a position in the image of the organ displayed by the display unit or related information related to the position, and a plurality of markers attached to a three-dimensional model of the organ And a marker detection device for detecting the position of the marker, wherein the position determination unit detects the position of the marker based on the detection result by the marker detection device. A three-dimensional model of an organ used in a medical observation support system for determining the position on a model and supporting observation of the organ in a subject, the single or plural criteria serving as a reference of correspondence by the correspondence control unit The position is predetermined, and the marker is attached to each reference position. According to the eleventh aspect of the invention, the predetermined instruction is made on the three-dimensional model of the organ created by the three-dimensional molding apparatus based on the three-dimensional image data of the organ obtained in advance, and the three-dimensional model of the organ A position determination unit that determines a position on the model indicated by the unit, a display unit that generates and displays an image of the organ based on three-dimensional image data of the organ, and the display unit displays the position on the model And the association control unit for correlating with the position in the image of the organ to be detected or the related information related to the position, the instruction unit using the three-dimensional model of the organ as an input interface The operation indicating the three-dimensional model of the organ can be reflected on the image of the organ displayed by the display unit. That is, it is possible to provide a medical observation support system which realizes highly practical observation support by intuitive operation. Further, according to the eleventh aspect of the present invention, the apparatus includes: a plurality of markers attached to a three-dimensional model of the organ; a marker attached to the instruction unit; and a marker detection device detecting a position of the marker. The position determination unit determines the position on the model based on the detection result by the marker detection device. In this way, it is possible to determine the position indicated by the pointing unit in the three-dimensional model of the organ in a practical manner. Further, according to the eleventh aspect of the present invention, the three-dimensional model of the organ created by the three-dimensional molding apparatus based on the three-dimensional image data of the organ obtained in advance and the three-dimensional model of the organ are predetermined. The position determination unit that determines the position on the model pointed by the instruction unit, the display unit that generates and displays the image of the organ based on the three-dimensional image data of the organ, and the position on the model A correspondence control unit that associates a position in an image of the displayed organ or related information related to the position, a plurality of markers attached to a three-dimensional model of the organ, and a marker attached to the indication unit; And a marker detection device for detecting the position of the marker, the position determination unit determining the position on the model based on the detection result by the marker detection device. A 3-dimensional model of an organ to be used in medical observation support system for supporting the observation of the organs, said reference comprising one or more reference positions of the correlation by the correlation controller predetermined in each reference position It is characterized in that the marker is attached. In this way, the position on the model can be associated with the position in the image of the organ displayed by the display unit in a practical manner.

It is a figure which illustrates the composition of the medical observation support system which is one example of the present invention. It is a functional block diagram explaining the principal part of an example of the control function with which the medical observation support system of FIG. 1 was equipped. The medical observation support system of FIG. 1 WHEREIN: It is a figure which roughly shows the aspect provided with a two-dimensional code as a marker, and an operator's finger tip corresponds to an indication part. The medical observation support system of FIG. 1 WHEREIN: The magnetic type position sensor is provided as a marker, and it is a figure which shows roughly the aspect to which the front-end | tip part of a pointer corresponds to an indication part. FIG. 2 is a view schematically showing an aspect in which a gyro sensor as an attitude sensor is attached to a three-dimensional model in the medical observation support system of FIG. 1; It is a figure which illustrates a mode that the gyro sensor and three-dimensional model shown in FIG. 5 were enclosed in the sterile bag member. It is a figure which shows typically a mode that the medical observation assistance system of FIG. 1 is actually used in the field of organ surgery. It is a flowchart explaining the principal part of an example of the observation assistance control of a present Example by CPU of the computer with which the medical observation assistance system of FIG. 1 was equipped. FIG. 7 is a view showing a specific example of a three-dimensional virtual image and a tomographic image displayed on the video display device in the medical observation support system of FIG. 1. It is the photograph which image | photographed one structural example of the three-dimensional model with which the medical observation assistance system of FIG. 1 was equipped.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating the configuration of a medical observation support system 10 (hereinafter simply referred to as the system 10) according to an embodiment of the present invention. As shown in FIG. 1, the system 10 of the present embodiment includes a computer 12 which is a main unit (computer), a position information acquiring device 14, and a three-dimensional model 16 of an organ to be observed. . Various forms of the position information acquisition device 14 are conceivable, and will be described later with reference to FIGS. The three-dimensional model 16 will be described later together with the description of the three-dimensional image data of the organ to be observed.

  The computer 12 includes a CPU 20 which is a central processing unit, a ROM 22 which is a read only memory, a RAM 24 which is an occasional write / read memory, a storage device 26, an input device 28, an image display device 30, and an input interface. And 32 are configured. The CPU 20 is a so-called microcomputer that processes and controls electronic information based on a predetermined program stored in advance in the ROM 22 while utilizing the temporary storage function of the RAM 24.

  The storage device 26 is a known storage device (storage medium) such as a hard disk drive capable of storing information. It may be a removable storage medium such as a small memory card or an optical disc. The input device 28 is a known input device such as a keyboard and a mouse. The image display device 30 is a known display device (display) such as a TFT (Thin Film Transistor Liquid Crystal). The input interface 32 is a known input interface that supplies a signal input from a position information acquisition device 14 described later to the CPU 20.

  FIG. 2 is a functional block diagram for explaining an essential part of an example of a control function provided in the system 10. As shown in FIG. Preferably, the image display unit 50, the position determination unit 62, the association control unit 64, and the distance calculation unit 66 shown in FIG. 2 are all functionally included in the CPU 20 of the computer 12. The respective control units may be individually provided in a plurality of CPUs or computers and configured to be able to communicate with each other.

  As shown in FIG. 2, an image database 68 is provided in the storage device 26 provided in the computer 12. The image database 68 stores three-dimensional image data of a subject obtained in advance by a known X-ray CT (Computed Tomography) apparatus, a magnetic resonance imaging (MRI; Magnetic Resonance Imaging) apparatus or the like. The three-dimensional image data includes at least three-dimensional image data of an organ to be observed by the system 10. In the three-dimensional image data obtained by X-ray CT, X-rays are irradiated to the subject from multiple directions, X-rays transmitted through the subject are detected by a detector, and transmitted X The information on the dose is processed by a computer and reconstructed as a three-dimensional image. The image database 68 stores, for example, three-dimensional gray-scale images of the subject obtained by X-ray CT, magnetic resonance apparatus, etc., in other words, pixel values (density values) for each voxel which is a unit volume of the subject. Be done.

  The three-dimensional model 16 is created by a known three-dimensional molding apparatus based on the three-dimensional image data of the organ stored in the image database 68. For example, based on three-dimensional image data of the organ, a three-dimensional organ model created so as to mimic the shape of the organ (that is, to have substantially the same shape) by a known three-dimensional printer (3D printer) (Replica). The three-dimensional model 16 is preferably a three-dimensional model of the liver created based on three-dimensional image data of the liver which is an organ to be observed. Preferably, it is made to have the same size (actual size) as the actual organ to be observed, but in consideration of ease of handling, etc., the size of the actual organ is enlarged or It may be created by reduction. In this case, the enlargement ratio or reduction ratio of the three-dimensional model 16 to the actual organ is input to the computer 12. Measurement results such as distance or length when the three-dimensional model 16 is used as an input interface are converted (converted) to the size of the actual organ based on the enlargement ratio or reduction ratio and the image display It is displayed on the device 30. Preferably, a lesion site such as a blood vessel or a tumor in an organ is made of a colored material, and surrounding tissues are made of a colorless transparent material. In the three-dimensional model 16, a plurality of (at least three) reference positions serving as a reference of correspondence by the correspondence control unit 64 described later are predetermined. For example, recesses (holes) of a predetermined depth are formed at the plurality of reference positions. When molding the three-dimensional model 16, markers described later may be integrally molded with the three-dimensional model 16 at each of the plurality of reference positions.

  The system 10 includes an instruction unit used to indicate the three-dimensional model 16. As the instruction unit, various modes can be considered such as the tip of the forceps, the fingertip of the practitioner (for example, the fingertip of the index finger), and the tip of the rod-like pointer (pointing device). That is, the instruction unit may be a part of a predetermined device or may be a part of a human body. For example, as described in detail below, a part to which a marker indicating the indication part is attached, or a part having a predetermined relationship with the position thereof corresponds to the indication part.

  Preferably, a plurality (at least three) of markers are attached to the three-dimensional model 16. For example, at each of a plurality of predetermined reference positions, markers indicating the respective reference positions are attached. Preferably, at least one marker is attached to the indicator. For example, a marker indicating the indication part is attached to the tip of the forceps defined as the indication part in advance, the fingertip of the practitioner, the tip of the bar-like pointer, or the like. The system 10 includes a marker detection device that detects the position of the marker. Hereafter, the specific example of the said instruction | indication part, the said marker, and the said marker detection apparatus is demonstrated using FIGS. 3-5 etc. FIG.

  FIG. 3 is a view schematically showing an aspect in which two-dimensional codes 34, 36 are provided as the markers in the system 10, and the finger tip 70 of the practitioner corresponds to the indication portion. FIG. 10 is a photograph of one configuration example of the three-dimensional model 16. In the embodiment shown in these figures, the three-dimensional model 16 is a portion 16a corresponding to a blood vessel made of a colored material, and a lesion site also made of a colored (preferably different color from the portion 16a) material. A corresponding portion 16b and a portion 16c corresponding to a surrounding tissue made of a transparent (preferably colorless and transparent) material are provided (the same as in FIGS. 4 and 5 used in the following description). A plurality of (three in FIG. 3) two-dimensional codes 34 are attached to the surface of the three-dimensional model 16 (portion 16c) to indicate that it is a reference position. For example, a two-dimensional code 34 printed in advance is attached to each of a plurality of reference positions predetermined in the three-dimensional model 16. The plurality of two-dimensional codes 34 are not particularly distinguished in FIG. 3, but are made to represent different information corresponding to the respective reference positions in the three-dimensional model 16 (markers of different codes) There is. A two-dimensional code 36 indicating that it is the instruction unit is attached to a fingertip (for example, a fingertip of the index finger) 70 of a practitioner who uses the system 10 to observe an organ. Preferably, a pre-printed two-dimensional code 36 is affixed to the fingertip 70 of the practitioner. The two-dimensional code 36 is different from at least the two-dimensional code 34 attached to the three-dimensional model 16. Here, in a mode that is sufficient if it is sufficient to determine the one-dimensional position (the position as a point) of the instruction unit, as shown in FIG. 3, if one two-dimensional code 36 is attached to the fingertip 70 of the practitioner. In a mode that determines the pointing direction (i.e., the pointing direction) of the pointing unit, it is preferable that two or more two-dimensional cords 36 be attached to the pointing unit. For example, a two-dimensional code 36 is attached to the fingertip 70 and the base of the finger of the practitioner, respectively. Alternatively, by determining the direction in advance with respect to the instruction unit and attaching the two-dimensional code 36, the direction indicated by the instruction unit can be determined from one two-dimensional code 36, and the two-dimensional code in the photographed image The distance from the imaging device 38 described later may be determined based on the size of the code 36 or the like. The two-dimensional code is, for example, an information expression code consisting of a two-dimensional (planar) graphic like a QR code (registered trademark) defined in Japanese Industrial Standard "JIS X 0510".

  In the embodiment shown in FIG. 3, the system 10 comprises an imaging device 38 for imaging the two-dimensional code 34,36. As this imaging device 38, a known digital camera that can input electronic data (image data) corresponding to a photographed image to the position information acquisition device 14 is used. Alternatively, an analog camera or the like provided with an A / D converter may be used. The imaging device 38 is connected to the position information acquisition device 14, and an image (video) captured by the imaging device 38 is input to the position information acquisition device 14. In FIG. 3, the aspect in which the imaging device 38 and the position information acquisition device 14 are configured as separate devices is illustrated, but the imaging device 38 is incorporated in the position information acquisition device 14 It may be The imaging device 38 may be built in the computer 12. The position information acquiring device 14 preferably comprises a known two-dimensional code reader for identifying the two-dimensional code 34, 36. The position determination unit 62 described in detail below may have a function as a two-dimensional code reader. In the aspect shown in FIG. 3, the imaging device 38 and the position information acquisition device 14 correspond to the marker detection device.

  The position determination unit 62 determines the position on the model indicated by the instruction unit in the three-dimensional model 16 based on the position information acquired by the position information acquisition device 14. The position on the model corresponds to the position on the surface (the surface of the portion 16c) of the three-dimensional model 16 and is preferably the coordinates of one point on the surface. Alternatively, not only the position on the three-dimensional model 16 but also any position on the image space where the three-dimensional model 16 is placed may be used. In addition to the position on the surface of the three-dimensional model 16, the direction of the pointing unit with respect to the three-dimensional model 16 (the relative direction in which the pointing unit points to the three-dimensional model 16) may be determined. The position determination unit 62 preferably determines the position on the model based on the detection result by the marker detection device. In this embodiment, the plurality of markers are arranged so that the marker detection device can always read three or more markers. In the aspect illustrated in FIG. 3, when the plurality of two-dimensional codes 34 are photographed by the imaging device 38 and read by the two-dimensional code reader provided in the position information acquisition device 14, the plurality of The position and direction of the two-dimensional code 34 are identified. Thereby, the positions and directions of the plurality of reference positions in the three-dimensional model 16 with respect to the imaging device 38 are determined. When the two-dimensional code 36 is photographed by the imaging device 38 and read by the two-dimensional code reader provided in the position information acquisition device 14, the position and direction of the two-dimensional code 36 with respect to the imaging device 38 are specified. Ru. Thereby, the position and the direction of the fingertip 70 of the practitioner with respect to the imaging device 38 are determined. From the position and direction with respect to the imaging device 38, the relative position of the fingertip 70 of the practitioner with respect to the three-dimensional model 16 is determined. That is, in the three-dimensional model 16, the position on the model indicated by the fingertip 70 of the practitioner is determined.

  In the aspect illustrated in FIG. 3, when the plurality of two-dimensional codes 34 are photographed by the imaging device 38 and read by the two-dimensional code reader provided in the position information acquisition device 14, the predetermined direction (for example, the vertical direction The attitude of the three-dimensional model 16 with reference to. For example, the posture of the three-dimensional model 16 based on the vertical direction is specified by the inclination of the plurality of reference positions predetermined in the three-dimensional model 16 with respect to the vertical direction. That is, in the mode shown in FIG. 3, the two-dimensional code 34, the imaging device 38, and the position information acquiring device 14 are used as posture sensors for detecting the posture of the three-dimensional model 16 based on a predetermined direction. Function.

  FIG. 4 is a view schematically showing an aspect in which magnetic position sensors 40 and 42 are provided as the markers in the system 10, and the tip end portion 72 of the pointer corresponds to the pointing portion. In the embodiment shown in FIG. 4, a plurality of (only one is shown in FIG. 4) magnetic position sensors 40 are attached to the surface of the three-dimensional model 16 (portion 16c) to indicate the reference position. . For example, the magnetic position sensor 40 is attached to each of a plurality of reference positions predetermined in the three-dimensional model 16. A magnetic position sensor 42 is attached to the tip end portion 72 of the pointer, which is a rod-like member. The magnetic position sensors 40 and 42 are connected to the position information acquisition device 14 by wire. Alternatively, they may be connected wirelessly. The magnetic position sensors 40 and 42 are known magnetic position sensors. For example, a transmitter (not shown) is provided separately from each of the magnetic position sensors 40 and 42, and a magnetic signal transmitted from the transmitter is By receiving each of the magnetic position sensors 40, 42, five-dimensional information, that is, three-dimensional position information and two-dimensional direction information, is obtained from each of the magnetic position sensors 40, 42 and supplied to the position information acquisition device 14. It is supposed to be In the aspect shown in FIG. 4, the position information acquisition device 14 corresponds to the marker detection device.

  In the aspect shown in FIG. 4, when position information and direction information corresponding to each of the plurality of magnetic position sensors 40 are supplied to the position information acquisition device 14, the plurality of magnetic position sensors 40 with respect to the transmitter are The position and direction are identified. Thereby, the positions and directions of the plurality of reference positions in the three-dimensional model 16 with respect to the transmitter are determined. When position information and direction information corresponding to the magnetic position sensor 42 are supplied to the position information acquisition device 14, the position and direction of the magnetic position sensor 42 with respect to the transmitter are specified. Thereby, the position and the direction of the tip portion 72 of the pointer with respect to the transmitter are determined. From the position and orientation with respect to the transmitter, the relative position of the tip 72 of the pointer with respect to the three-dimensional model 16 is determined. That is, in the three-dimensional model 16, the position on the model indicated by the tip portion 72 of the pointer is determined.

  In the aspect shown in FIG. 4, when position information and direction information corresponding to each of the plurality of magnetic type position sensors 40 are supplied to the position information acquiring device 14, a predetermined direction (for example, the vertical direction) is used as a reference. The attitude of the three-dimensional model 16 is identified. That is, in the aspect shown in FIG. 4, the magnetic position sensor 40 and the position information acquisition device 14 function as an attitude sensor that detects an attitude of the three-dimensional model 16 with reference to a predetermined direction.

  The system 10 may include an optical marker as the marker. For example, in the aspect described above with reference to FIG. 3 or FIG. 4, the two-dimensional code 34, 36 or the magnetic position sensor 40, 42 may be replaced by an optical marker. As the optical marker, a light emitting element such as an LED that emits near infrared light, a reflection marker that reflects light, or the like is suitably used. In such an embodiment, a marker detection device for detecting the position and direction of the optical marker, such as a near infrared camera for detecting near infrared light, is provided. The position and the direction of each optical marker detected by the marker detection device are supplied to the position information acquisition device 14, and the relative position of the pointing unit with respect to the three-dimensional model 16 is determined by the same method as described above. Be done. That is, in the three-dimensional model 16, the position on the model indicated by the instruction unit is determined.

  FIG. 5 is a view schematically showing an aspect in which a gyro sensor 44 as an attitude sensor is attached to the three-dimensional model 16 in the system 10. In the embodiment shown in FIG. 5, a gyro sensor 44 as an attitude sensor is attached to the three-dimensional model 16. The gyro sensor 44 is a known gyro sensor that detects an angle and a direction with respect to a predetermined direction (for example, the vertical direction). In the aspect illustrated in FIG. 5, the position information acquisition device 14 functions as a receiving unit that wirelessly receives the detection result of the gyro sensor 44. When the detection result by the gyro sensor 44 is supplied to the position information acquiring device 14, the attitude of the three-dimensional model 16 with reference to the predetermined direction is specified. In the embodiment shown in FIG. 5, another marker is attached to the three-dimensional model 16 in order to determine the relative position of the pointing unit with respect to the three-dimensional model 16. For example, the two-dimensional code 34 described above with reference to FIG. 3 or the magnetic position sensor 40 or the like described above with reference to FIG. 4 is attached to the three-dimensional model 16. Here, in the aspect in which the three-dimensional model 16 includes the gyro sensor 44, the number of markers for the three-dimensional model 16 is smaller than that in the aspect described above with reference to FIG. 3 or FIG. The relative position can be determined.

  As mentioned above, although the specific example of the said instruction | indication part in the said system 10, the said marker, and the said marker detection apparatus was demonstrated using FIGS. 3-5, another aspect is also considered. For example, a contact-type position sensor is provided on the surface of the three-dimensional model 16 (that is, the surface of the portion 16c), and the contact position of the pointing unit with respect to the three-dimensional model 16 is detected by the contact-type position sensor. May be In such an embodiment, the relative position of the indicator with respect to the three-dimensional model 16 is determined based on the detection result of the contact-type position sensor. That is, in the three-dimensional model 16, the position on the model indicated by the instruction unit is determined. Alternatively, the indication unit for the three-dimensional model 16 is provided by an ultrasonic detection device that includes an ultrasonic transmitter, receives ultrasonic waves emitted by the ultrasonic transmitter with a plurality of sensors, and detects the position according to the time difference. The relative position of may be determined. Alternatively, an RFID (Radio Frequency Identification) system provided with an active type wireless tag that emits radio waves may be used as the marker and the marker detection device.

  FIG. 6 is a view illustrating how the three-dimensional model 16 to which the gyro sensor 44 is attached is enclosed in a bag member 74. As shown in FIG. The system 10 preferably comprises a sterile bag member 74. The bag member 74 is a colorless and transparent bag made of a vinyl resin or the like, and has been subjected to a sterilization treatment to such an extent that it can be brought into the operating room. The three-dimensional model 16 and the gyro sensor 44 are brought into the operating room in a state of being enclosed in the bag member 74 and used for observation support in the system 10. After the three-dimensional model 16 and the gyro sensor 44 are sealed in the bag member 74, the three-dimensional model 16 and the gyro sensor 44 may be used in a so-called vacuum pack state in which the air inside the bag member 74 is evacuated. The three-dimensional model 16 to which the two-dimensional code 34 described above with reference to FIG. 3 or the magnetic position sensor 40 or the like described above with FIG. 4 is attached corresponds to the two-dimensional code 34 or the magnetic position sensor 40. And may be used in the state of being enclosed in the bag member 74.

  The image display unit 50 causes the video display device 30 to display various images (videos) related to control by the system 10 through a graphics processor (not shown) or the like. The image display unit 50 generates an image of the organ based on the three-dimensional image data of the organ stored in the image database 68 and causes the image display device 30 to display the image. In order to perform such processing, as shown in FIG. 2, the image display unit 50 is configured to display a three-dimensional virtual image display unit 52, a tomographic image display unit 54, an indication position display unit 56, an incision line display unit 58, and a distance display. A unit 60 is provided.

  The three-dimensional virtual image display unit 52 generates a three-dimensional virtual image of the organ based on the three-dimensional image data of the organ stored in the image database 68 and causes the video display device 30 to display the image. The three-dimensional virtual image of the organ is an image showing the shape of the entire organ, and preferably is an external image of the organ. That is, preferably, the three-dimensional virtual image display unit 52 displays a pictorial image showing the external shape of the organ (for example, an image showing the appearance viewed from an arbitrary direction) based on the three-dimensional image data. That is, the three-dimensional image data previously obtained corresponding to the organ to be observed by the system 10 and stored in the image database 68 is read, for example, by a known technique such as volume rendering method or surface rendering method The external image of the organ is generated based on two-dimensional image data and displayed on the image display device 30. FIGS. 3 to 5 and 7 illustrate an example in which the three-dimensional virtual image 80 of the organ generated based on the three-dimensional image data of the organ is displayed on the video display device 30.

  The tomographic image display unit 54 generates a tomographic image of the organ at a designated position or cutting line based on the three-dimensional image data of the organ stored in the image database 68 and causes the image display device 30 to generate the tomographic image. Display. The tomographic image of the organ is an image showing a cross section of the organ. A tomographic image obtained in advance by an X-ray CT apparatus or the like may be displayed as it is, or an image corresponding to a virtual cross section generated based on three-dimensional image data may be used. That is, the tomographic image display unit 54 reads out three-dimensional image data obtained in advance corresponding to an organ to be observed by the system 10 and stored in the image database 68, and the three-dimensional image is read by a known technique. Based on the data, an image showing a cross section of the organ is generated and displayed on the image display device 30. FIG. 7 exemplifies a state in which the tomographic image 86 of the organ generated based on the three-dimensional image data of the organ is displayed on the image display device 30.

  The association control unit 64 associates the position on the model indicated by the instruction unit in the three-dimensional model 16 with the position in the image of the organ displayed on the image display device 30 or the related information related to the position. That is, coordinate system registration processing is performed to associate the coordinate system corresponding to the three-dimensional model 16 and the instruction unit with the coordinate system corresponding to the image of the organ displayed on the image display device 30. The positional relationship between the feature points is matched between the three-dimensional model 16 and the image of the organ displayed on the image display device 30 based on a plurality of feature points in the organ, not the correspondence of the coordinate system. The association may be performed. Preferably, the association control unit 64 associates the position on the model with the position in the three-dimensional virtual image 80 of the organ displayed by the three-dimensional virtual image display unit 52. Preferably, the position on the model is associated with the position in the tomographic image 86 of the organ displayed by the tomographic image display unit 54. Hereinafter, an example of a specific algorithm related to the association by the association control unit 64 will be described.

The correspondence (coordinate system registration) between the coordinate system corresponding to the instruction unit and the coordinate system corresponding to the image of the organ in the system 10 will be described. A coordinate system corresponding to the instruction unit is L, a coordinate system corresponding to the marker attached to the instruction unit is M, and a marker detection device for detecting the marker (for example, the position information acquisition device 14 or the imaging device 38) Let P be a coordinate system corresponding to Y and I be a coordinate system corresponding to the image of the organ. For example, as shown in FIG. 3, when the marker of the instruction unit and the marker of the three-dimensional model 16 are photographed by an imaging device common to each other, the coordinate systems L and M of both are common. When the coordinate system L and the coordinate system M are substantially the same, the conversion from the coordinate system L to the coordinate system M described below may not be performed. The transformation matrix from the coordinate system L to the coordinate system M is T _ML , the transformation matrix from the coordinate system M to the coordinate system P is T _PM , and the transformation matrix from the coordinate system P to the coordinate system I T _IP When the coordinates p _L in the coordinate system L corresponding to the instruction unit, as shown in the following equation (1), corresponding to an image of the organ using the transformation matrix T _ML, T _PM, T _IP It is converted into the coordinate p _I in the coordinate system I. That is, the coordinates p _L in the coordinate system L are associated by the following equation (1) in the coordinate p _I in the coordinate system I. Coordinate transformation of a rigid body takes rotation and translation as elements. Therefore, the transformation of the coordinate p by the transformation matrix T is expressed as the following equation (2) using the rotation matrix R and the transformation vector t.
p _I = T _IP T _PM T _ML p _L (1)
Tp = Rp + t (2)

In the system 10, the position and the direction (for example, the position and the direction of the tip of the pointer) corresponding to the pointing portion, and the position and the direction of the marker attached to the pointing portion (for example, the attachment position of the marker in the pointer The relationship with the direction is predetermined. This relationship corresponds to the transformation matrix T _ML from the coordinate system L to the coordinate system M. Information corresponding to the position and direction of the marker attached to the instruction unit is acquired by the marker detection device. The transformation matrix T _PM from the coordinate system M to the coordinate system P is determined by this information. That is, by detecting the information corresponding to the position and the direction of the marker attached to the pointing unit by the marker detection device, the coordinate p _L in the coordinate system L corresponding to the pointing portion corresponds to the marker detection device It can be converted into the coordinate p _P in the coordinate system P of. Coordinate p _P in the coordinate system P corresponding to the marker detecting device, as shown in the following equation (3), the coordinates p _I in a coordinate system I corresponding to the image of the organ using the transformation matrix T _IP It is converted.
p _I = T _IP p _P (3)

The conversion from the coordinate system P corresponding to the marker detection device to the coordinate system I corresponding to the image of the organ is realized, for example, by matching the reference positions in the respective coordinate systems. Preferably, in the coordinate system I corresponding to the image of the organ, a reference position relating to the correspondence with the coordinate system L corresponding to the instruction unit is predetermined. The reference position is preferably any of the reference positions defined in the three-dimensional model 16. In the three-dimensional model 16, when the pointing unit points to the reference position, a reference position on the coordinate system P corresponding to the marker detection device, and a reference position on the coordinate system I corresponding to the image of the organ by performing mapping to match the transformation matrix T _IP to the coordinate system I is determined from the coordinate system P. For example, assuming the coordinates of the reference position in the coordinate system P corresponding to the marker detection device as pi, the coordinates of the reference position in the coordinate system I corresponding to the image of the organ as qi, and the number of reference positions as N (4) ) is the transformation matrix T ^* which satisfies the equation, corresponding to the transformation matrix T _IP.

  Although the correspondence between the coordinate system L corresponding to the instruction unit and the coordinate system I corresponding to the image of the organ has been described above, the coordinate system corresponding to the three-dimensional model 16 and the coordinates corresponding to the image of the organ The correspondence with the system I is also realized by the same algorithm as described above. That is, according to the same algorithm as described above, a plurality of reference positions predetermined in the three-dimensional model 16 and a plurality of reference positions predetermined in the coordinate system I corresponding to the image of the organ It is realized by making L and I coincide with each other. A coordinate system L corresponding to the instruction unit is associated with a coordinate system I corresponding to the image of the organ, and a coordinate system corresponding to the three-dimensional model 16 and a coordinate system I corresponding to the image of the organ By being correlated, the position on the model indicated by the instruction unit in the three-dimensional model 16 can be correlated with the image of the organ displayed by the image display unit 50.

  The distance calculation unit 66 is configured on the model indicated by the instruction unit in the three-dimensional model 16 in the image of the organ displayed by the image display unit 50 based on the result of the association by the association control unit 64. Calculate the distance corresponding to the position. Specifically, the distance is calculated based on three-dimensional image data of the organ stored in the image database 68. Preferably, the distance calculation unit 66 calculates the distance (length of blood vessel) of the main blood vessel indicated by the pointing unit in the three-dimensional model 16 in the image of the organ. The main blood vessel is an artery or a vein present in the direction (inside of the organ) pointed by the indicator, and is, for example, a blood vessel generally anatomically named. Preferably, the distance calculation unit 66 calculates the distance between the position on the model indicated by the instruction unit in the three-dimensional model 16 and the lesion site in the organ in the image of the organ. Preferably, the distance calculation unit 66 sets a distance between a plurality of on-model positions (for example, two points indicated in succession) indicated in the three-dimensional model 16 in the image of the organ. calculate.

  The image display unit 50 reflects the position on the model indicated by the instruction unit in the three-dimensional model 16 on the image of the organ based on the result of the association by the association control unit 64. That is, the image of the organ displayed on the image display device 30 is changed according to the change in the position on the model indicated by the instruction unit in the three-dimensional model 16. Alternatively, an additional image (including text such as a numerical value) is combined and displayed on the image of the organ displayed on the image display device 30. In this case, the additional image or the like corresponds to the related information related to the position.

  The distance display unit 60 causes the video display device 30 to display the distance calculated by the distance calculation unit 66. Preferably, a numerical value corresponding to the distance calculated by the distance calculation unit 66 is combined and displayed on the image of the organ displayed on the image display device 30. In the aspect in which the distance calculation unit 66 calculates the length of the blood vessel pointed by the instruction unit in the three-dimensional model 16, a numerical value indicating the length of the blood vessel as well as indicating the target blood vessel in the image of the organ Make a composite display. In the aspect in which the distance calculation unit 66 calculates the distance between the position on the model indicated by the instruction unit in the three-dimensional model 16 and the lesion site in the organ, the target lesion site in the image of the organ is shown. , Synthetically display numerical values representing the distances. In an aspect in which the distance calculation unit 66 calculates the distance between a plurality of positions on the model indicated by the instruction unit in the three-dimensional model 16, the distance calculation unit 66 indicates a plurality of positions indicated by the instruction unit in the image of the organ. , Synthetically display numerical values representing the distances. In this case, the numerical value corresponds to the related information related to the position.

  The three-dimensional virtual image display unit 52 displays the three-dimensional image displayed on the video display device 30 based on the posture information of the three-dimensional model 16 acquired by the position information acquisition device 14 or the gyro sensor 44 or the like. The attitude of the virtual image 80 is changed. That is, the attitude of the three-dimensional model 16 based on a predetermined direction (for example, the vertical direction) determined by the position determination unit 62 based on the detection result of the position information acquisition device 14 or the gyro sensor 44 or the like. The posture (direction and angle) of the three-dimensional virtual image 80 displayed on the video display device 30 is changed based on the change of.

  The incision line display unit 58 combines the incision line 82 corresponding to the transition of the position on the model indicated by the instruction unit in the three-dimensional model 16 on the three-dimensional virtual image 80 displayed on the image display device 30. Let For example, in the embodiment shown in FIG. 3, when the finger tip 70 of the practitioner as the indication part has shifted from the position shown by the broken line to the position shown by the solid line (tracked on the surface of the three-dimensional model 16), its locus is 3 It will be shown by a thick broken line on the dimensional model 16. In this case, as shown in FIG. 3, an incision line 82 corresponding to the locus is generated and displayed on the three-dimensional virtual image 80 in combination. In this case, the cutting line 82 corresponds to the related information related to the position. Alternatively, a virtual tomographic image showing a cross section (cut surface) of the organ, which is obtained by cutting the three-dimensional virtual image 80 at a plane corresponding to the cut line 82, is displayed on the video display device 30. May be

  FIG. 7 is a view schematically showing how the system 10 is actually used at the surgical site of the organ. In FIG. 7, the surgeon 92 involved in the operation of the liver 90a in the subject (subject) 90 holds the three-dimensional model 16 created based on the three-dimensional image data of the liver 90a in his hand, The aspect which points the three-dimensional model 16 by the part 76 is illustrated. That is, in the mode shown in FIG. 7, the distal end portion 76 of the forceps corresponds to the instruction portion. The assistant 94 and the nurse 96 involved in the operation are watching the display of the image display 30 while listening to the explanation of the operator 92. The three-dimensional virtual image 80 and the tomographic image 86 generated on the basis of three-dimensional image data of the liver 90 a are displayed on the image display device 30.

  The indication position display unit 56 reflects the position on the model indicated by the indication unit in the three-dimensional model 16 on the three-dimensional virtual image 80. For example, in the mode shown in FIG. 4, the mark 84 representing the position in the three-dimensional virtual image 80 corresponding to the position on the model pointed by the tip portion 72 of the pointer as the pointing unit It is composited on the front side. In the embodiment shown in FIG. 7, the front side of the three-dimensional virtual image 80 is a mark 84 representing the position in the three-dimensional virtual image 80 corresponding to the position on the model pointed by the tip end portion 76 of the forceps as the pointing unit. Is displayed on the screen. In this case, the mark 84 corresponds to the related information related to the position. In FIG. 4 and FIG. 7, the mark 84 is indicated by an asterisk. The designated position display unit 56 changes the relative position of the mark 84 with respect to the three-dimensional virtual image 80 based on the change in the position on the model determined by the position determination unit 62.

  The tomographic image display unit 54 displays a tomographic image 86 corresponding to a cross section including the position on the model indicated by the instruction unit in the three-dimensional model 16. That is, among the tomographic images 86 that can be displayed based on the three-dimensional image data of the organ, the tomographic image 86 corresponding to the cross section including the position on the model is selected and displayed. The indication position display unit 56 reflects the position on the model indicated by the indication unit in the three-dimensional model 16 on the tomographic image 86. For example, in the embodiment shown in FIG. 7, the mark 88 representing the position in the tomographic image 86 corresponding to the position on the model pointed by the tip portion 76 of the forceps as the pointing portion is synthesized on the front side of the tomographic image 86 It is displayed. In FIG. 7, the mark 88 is indicated by an asterisk. The designated position display unit 56 changes the relative position of the mark 88 with respect to the tomographic image 86 based on the change in the position on the model determined by the position determination unit 62.

  FIG. 9 is a view showing a specific example of the three-dimensional virtual image 80 and the tomographic image 86 displayed on the video display device 30 by the image display unit 50. As shown in FIG. In the example shown in FIG. 9, the lesion site (tumor or the like) of the target organ is shown by the site 80 b in the three-dimensional virtual image 80 and by the site 86 b in the tomographic image 86. In the three-dimensional model 16, an incision line 82 corresponding to the transition of the position on the model indicated by the instruction unit is synthetically displayed on the front side of the three-dimensional virtual image 80. Furthermore, a pointer image 98 indicating the instruction section is composited and displayed on the front side of the three-dimensional virtual image 80. The position of the pointer image 98 with respect to the three-dimensional virtual image 80 is changed according to the transition of the position on the model indicated by the instruction unit in the three-dimensional model 16. In the three-dimensional model 16, an x-coordinate line 100x and a y-coordinate line 100y representing a position on the model indicated by the instruction unit are synthetically displayed on the front side of the tomographic image 86. The x coordinate line 100x and the y coordinate line 100y respectively correspond to the x coordinate and y coordinate of the position on the model in the plane corresponding to the tomographic image 86, and the intersection thereof corresponds to the position on the model. Furthermore, in the example shown in FIG. 9, the text 102 indicating the xy coordinates (x1, y1) of the position on the model in the plane corresponding to the tomographic image 86 is synthesized and displayed on the front side of the tomographic image 86 There is.

  As described above, the system 10 of this embodiment uses the three-dimensional model 16 as an input interface, and interacts with the three-dimensional virtual image 80 and the tomographic image 86 etc. displayed on the video display device 30. Make it possible. The three-dimensional model 16, the three-dimensional virtual image 80, and the tomographic image 86 are organ models in which the three-dimensional image data are embodied in different modes based on common three-dimensional image data. That is, the three-dimensional model 16 is a realized organ model having the same shape (entity) as the organ, which is embodied so as to allow the three-dimensional image data to be tactilely confirmed. The three-dimensional virtual image 80 and the tomographic image 86 are virtualized organ models that have the same shape information and detailed structure information as the organ, which can visually confirm the three-dimensional image data. By synchronizing the display of the three-dimensional virtual image 80 as the virtualized organ model, the tomographic image 86 and the like with the operation using the three-dimensional model 16 as the actualized organ model, the organ in the real space It is possible to visually recognize the distance, the virtual cross section, and the like of each part in the organ on the virtual space while confirming the shape of the object by touch. That is, the merits of the actualized organ model and the virtualized organ model can be compatible. At the same time, by utilizing the three-dimensional model 16 itself as the actualized organ model as an operation (input) device, it is possible to construct a navigation system that realizes appropriate surgical support by intuitive operation.

  FIG. 8 is a flow chart for explaining the main part of one example of the observation support control of the present embodiment by the CPU 20 of the computer 12 and is repeatedly executed.

  First, in step (hereinafter, step will be omitted) S1, three-dimensional image data of an organ to be observed is read out from the image database 68. Next, at S2, based on the three-dimensional image data read at S1, a three-dimensional virtual image 80 and a tomographic image 86 of the organ are generated and displayed on the video display device 30. Next, in S3, information indicating the positions and directions of the three-dimensional model 16 and the instruction unit is acquired by the position information acquisition device 14, and the position on the model indicated by the instruction unit in the three-dimensional model 16 is determined. Ru. Next, in S4, an associating process is performed to associate the position on the model determined in S3 with the position in the three-dimensional virtual image 80 and the tomographic image 86 of the organ displayed in S2, that is, coordinate system registration is performed. .

  Next, in S5, it is determined whether the three-dimensional model 16 has been moved. Specifically, it is determined whether the attitude of the three-dimensional model 16 based on the predetermined direction has been changed. When the determination of S5 is negative, the processing of S10 and the subsequent steps are executed. However, when the determination of S5 is positive, the three-dimensional model acquired by the position information acquiring device 14 in S6. The displacement of the three-dimensional model 16 with respect to the predetermined direction is calculated based on sixteen pieces of posture information. Next, in S7, the three-dimensional virtual image 80 is rotated based on the displacement calculated in S6. Next, in S8, 3D rendering is performed based on the rotation result of S7. That is, the three-dimensional virtual image 80 (a three-dimensional virtual image 80 corresponding to a posture rotated based on the displacement calculated at S6) as a result of the rotation at S7 is generated. Next, after the three-dimensional virtual image 80 generated at S8 is displayed on the video display device 30 at S9, this routine is ended.

  In S10, it is determined whether or not the input of the designated position, that is, the input of the position on the model indicated by the instruction unit in the three-dimensional model 16 is performed. If the determination in S10 is negative, the processing in S14 and the subsequent steps is executed. However, if the determination in S10 is positive, in S11, the distance corresponding to the input model position is the above It is calculated in the three-dimensional virtual image 80 (tomographic image 86) of the organ. Next, in S12, the distance calculated in S11 is displayed on the video display device 30. Next, in S13, after the input position on the model is combined and displayed on the three-dimensional virtual image 80 and the tomographic image 86 displayed on the image display device 30, this routine is ended with that. . In S14, it is determined whether or not an incision line has been input according to the transition of the position on the model indicated by the instruction unit in the three-dimensional model 16. If the determination in S14 is negative, the present routine is ended with that, but if the determination in S14 is affirmed, in S15, an incision simulation based on the incision line based on the transition of the position on the model is performed. To be executed. For example, a virtual tomographic image showing a cross section (cut surface) of the organ obtained by cutting the three-dimensional virtual image 80 at a plane corresponding to the cut line is generated. Next, in S16, the incision line 82 due to the transition of the position on the model is combined with the three-dimensional virtual image 80, and the result of the incision simulation in S15 is displayed on the image display device 30. The routine is terminated.

  In the above control, S2, S8, S9, S12, S13 and S16 are for the operation of the image display unit 50, S2 and S7 to S9 are for the operation of the three-dimensional virtual image display unit 52, and S2 is the tomographic image display In the operation of the unit 54, S13 is the operation of the pointing position display unit 56, S15 and S16 are the operation of the incision line display unit 58, S12 is the operation of the distance display unit 60, and S3 and S6 are the position determination S4 corresponds to the operation of the association control unit 64, and S11 corresponds to the operation of the distance calculation unit 66.

  According to the present embodiment, the three-dimensional model 16 of the organ created by the three-dimensional molding apparatus based on the three-dimensional image data of the organ obtained in advance and the fingertip 70 of the practitioner in the three-dimensional model 16. An image of the organ based on the three-dimensional image data of the organ, the position determination unit 62 (S3 and S6) which determines the position on the model pointed by the instruction unit such as the tip end portion 72 of the pointer and the tip end 76 of the forceps Image display unit 50 (S2, S8, S9, S12, S13, S16) for generating and displaying the image, and the position on the model at or in the position of the image of the organ displayed by the image display unit 50. Since the correspondence control unit 64 (S4) to be associated with the related related information is provided, the instruction unit uses the three-dimensional model 16 of the organ as an input interface. An operation of pointing to 3D model 16 of the vessel, can be reflected in the image of the organ to be displayed by the image display unit 50. That is, it is possible to provide the system 10 which realizes highly practical observation support by intuitive operation.

  Since the distance calculating unit 66 (S11) for calculating the distance corresponding to the position on the model in the image of the organ is provided, the three-dimensional model 16 of the organ is used as an input interface to The distance corresponding to the operation can be easily calculated.

  The image display unit 50 generates and displays a three-dimensional virtual image 80 of the organ on the basis of three-dimensional image data of the organ, and an incision line 82 corresponding to the transition of the position on the model is Since the composite display is performed on the three-dimensional virtual image 80 of the organ, the three-dimensional model 16 of the organ is used as an input interface, according to the operation of the instruction unit on the three-dimensional virtual image 80 of the organ The incision line 82 can be displayed in a simple manner.

  The image display unit 50 generates and displays a three-dimensional virtual image 80 of the organ based on three-dimensional image data of the organ, and the position on the model is the three-dimensional virtual image 80 of the organ. Since the three-dimensional model 16 of the organ is used as an input interface, the position indicated by the instruction unit can be simply synthesized and displayed on the three-dimensional virtual image 80 of the organ.

  The image display unit 50 generates and displays a tomographic image 86 of the organ based on three-dimensional image data of the organ, and compositely displays the position on the model on the tomographic image 86 of the organ. Since the three-dimensional model 16 of the organ is used as an input interface, the position indicated by the pointing unit can be simply synthesized and displayed on the tomographic image 86 of the organ.

  A gyro sensor 44 or the like as a posture sensor for detecting a posture of the three-dimensional model 16 of the organ with reference to a predetermined direction is provided, and the image display unit 50 determines the organ based on three-dimensional image data of the organ. The three-dimensional virtual image 80 of the organ is generated and displayed, and the posture of the three-dimensional virtual image 80 of the organ is changed based on the detection result by the posture sensor. The display control can be realized such that the posture of the three-dimensional virtual image 80 of the organ follows the posture of the three-dimensional model 16 of the organ by using 16 as an input interface.

  Since the three-dimensional model 16 of the organ and the posture sensor are enclosed in the bag member 74, the sterile three-dimensional model 16 of the organ is provided with a sterile bag member 74. It can be used as

  The marker includes a plurality of markers attached to the three-dimensional model 16 of the organ, a marker attached to the instruction unit, and a marker detection device for detecting the position of the marker, and the position determination unit 62 Since the position on the model is determined based on the detection result by the detection device, the position indicated by the pointing unit in the three-dimensional model 16 of the organ can be determined in a practical manner.

  Since the marker detection device is the magnetic position sensors 40 and 42 that magnetically detect the marker, it is possible to determine the position indicated by the indicator in the three-dimensional model 16 of the organ in a practical manner.

  The marker is an optical marker, and the marker detection device is a device for optically detecting the marker. Therefore, in a practical mode, the position indicated by the indicator in the three-dimensional model 16 of the organ It can be determined.

  The marker is a two-dimensional code 34, 36, and the marker detection device is the imaging device 38 and the position information acquisition device 14 as a two-dimensional code reader. It is possible to determine the position indicated by the instruction unit in the model 16.

  A three-dimensional model 16 of an organ used in the system 10, wherein one or more reference positions serving as the reference of the correspondence by the association control unit 64 are predetermined, and the markers are attached to the respective reference positions Since it is an object, the position on the model indicated by the instruction unit in the three-dimensional model 16 of the organ can be associated with the position in the image of the organ displayed by the image display unit 50 in a practical mode.

  Although the preferred embodiments of the present invention have been described in detail with reference to the drawings, the present invention is not limited to these embodiments, and various modifications may be made without departing from the scope of the present invention. It will be implemented. For example, the related information related to the position is not limited to the related information in the embodiment, and various related information other than the above can be displayed in various manners.

  10: medical observation support system, 14: position information acquisition device (marker detection device), 16: three-dimensional model, 34, 36: two-dimensional code (marker), 38: imaging device, 40, 42: magnetic position sensor ( Markers) 44: gyro sensor (posture sensor) 50: image display unit 62: position determination unit 64: association control unit 66: distance calculation unit 70: practitioner's finger (instruction unit) 72: Tip portion (pointing portion) of pointer 74: Bag member 76: Tip portion (pointing portion) of forceps 80: Three-dimensional virtual image 82: Incised line 86: Tomographic image 90: Subject (subject), 90a: Liver (Organ)

Claims (11)

A medical observation support system for supporting observation of an organ in a subject, comprising:
A three-dimensional model of the organ created by a three-dimensional molding apparatus based on three-dimensional image data of the organ obtained in advance;
A position determination unit that determines a position on the model indicated by a predetermined instruction unit in the three-dimensional model of the organ;
A display unit that generates and displays an image of the organ based on three-dimensional image data of the organ;
A correspondence control unit that associates the position on the model with a position in the image of the organ displayed by the display unit or related information related to the position ;
A medical observation support system , comprising: a distance calculation unit which calculates a distance corresponding to the position on the model in the image of the organ . A medical observation support system for supporting observation of an organ in a subject, comprising:
A three-dimensional model of the organ created by a three-dimensional molding apparatus based on three-dimensional image data of the organ obtained in advance;
A position determination unit that determines a position on the model indicated by a predetermined instruction unit in the three-dimensional model of the organ;
A display unit that generates and displays an image of the organ based on three-dimensional image data of the organ;
A correspondence control unit that associates the position on the model with a position in the image of the organ displayed by the display unit or related information related to the position;
Have,
The display unit is
A three-dimensional virtual image of the organ is generated and displayed based on the three-dimensional image data of the organ.
An incision line corresponding to the transition of the position on the model is displayed in combination on a three-dimensional virtual image of the organ.
Medical observation support system characterized by The display unit is
A three-dimensional virtual image of the organ is generated and displayed based on the three-dimensional image data of the organ.
The medical observation support system according to claim 1 or 2 , wherein the position on the model is combined and displayed on a three-dimensional virtual image of the organ. The display unit is
The tomographic image of the organ is generated and displayed based on the three-dimensional image data of the organ.
The medical observation support system according to any one of claims 1 to 3 , wherein the position on the model is synthetically displayed on a tomographic image of the organ. A posture sensor for detecting a posture of the three-dimensional model of the organ with reference to a predetermined direction;
The display unit is
A three-dimensional virtual image of the organ is generated and displayed based on the three-dimensional image data of the organ.
The medical observation support system according to any one of claims 1 to 4 , wherein the posture of the three-dimensional virtual image of the organ is changed based on the detection result by the posture sensor. Equipped with sterile bag members,
The medical observation support system according to claim 5 , wherein the three-dimensional model of the organ and the posture sensor are enclosed in the bag member. A plurality of markers attached to the three-dimensional model of the organ;
A marker attached to the indicator;
A marker detection device for detecting the position of the marker;
The medical observation support system according to any one of claims 1 to 6 , wherein the position determination unit determines the position on the model based on a detection result by the marker detection device. The marker is a magnetic marker,
The medical observation support system according to claim 7 , wherein the marker detection device is a device that magnetically detects the marker. The marker is an optical marker,
The medical observation support system according to claim 7 , wherein the marker detection device is a device that optically detects the marker. A medical observation support system for supporting observation of an organ in a subject, comprising:
A three-dimensional model of the organ created by a three-dimensional molding apparatus based on three-dimensional image data of the organ obtained in advance;
A position determination unit that determines a position on the model indicated by a predetermined instruction unit in the three-dimensional model of the organ;
A display unit that generates and displays an image of the organ based on three-dimensional image data of the organ;
A correspondence control unit that associates the position on the model with a position in the image of the organ displayed by the display unit or related information related to the position;
A plurality of markers attached to the three-dimensional model of the organ;
A marker attached to the indicator;
A marker detection device for detecting the position of the marker;
Have,
The position determination unit determines the position on the model based on a detection result by the marker detection device,
The marker is a two-dimensional code,
The marker detection device is an imaging device and a two-dimensional code reader
Medical observation support system characterized by A three-dimensional model of the organ created by a three-dimensional molding apparatus based on three-dimensional image data of the organ obtained in advance;
A position determination unit that determines a position on the model indicated by a predetermined instruction unit in the three-dimensional model of the organ;
A display unit that generates and displays an image of the organ based on three-dimensional image data of the organ;
A correspondence control unit that associates the position on the model with a position in the image of the organ displayed by the display unit or related information related to the position;
A plurality of markers attached to the three-dimensional model of the organ;
A marker attached to the indicator;
A marker detection device for detecting the position of the marker;
Have,
The position determination unit is a three-dimensional model of an organ used in a medical observation support system that supports observation of the organ in a subject, which determines the position on the model based on a detection result by the marker detection device. ,
A three-dimensional model of an organ, wherein a single or a plurality of reference positions as a reference of the correspondence by the correspondence control unit are predetermined and the marker is attached to each reference position.