JP2008264520A - Apparatus, method, and program for displaying surgery support information - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately indicate that a part imaged by an endoscope corresponds to which part of a patient without using a new special endoscope. <P>SOLUTION: An apparatus 1 for displaying surgery support information includes: the endoscope 11; marker balls 12 positioned and fixed to the endoscope 11; a CT device 30 for acquiring information which indicates the shape of the patient 60; an imaging device 20 for imaging the surface of the patient 60 and the marker balls 12; a surface shape calculating part 43 for calculating information, which indicates the shape of the surface of the patient 60, from an image; a coordinate axis coincidence part 44 for matching the shape of the surface of the patient by the CT device 30 with the shape of the surface of the patient calculated based on the image; an endoscope optical axis calculating part 45 for calculating the coordinates of the marker balls 12, based on the image, and calculating a half line which indicates the imaging directional optical axis of the endoscope 11; an intersection point calculating part 46 for calculating intersection points between the half line and a surface constituting the inner part of the patient 60; and an output part 47 for superimposing the information which indicates shape of the patient 60 onto information which indicates the intersections, and outputting them. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a surgery support information display apparatus, a surgery support information display method, and a surgery support information display program that provide information about an image captured by an endoscope to an operator or the like.

  Conventionally, when a surgical instrument such as an endoscope is inserted into the body of a patient, the exact position of the distal end of the surgical instrument is obtained by CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) taken before surgery. Surgery navigation (surgery support information display) is performed on the image to assist the surgeon. However, each of these methods only displays the position of the distal end of a surgical instrument or pointer, such as a surgical instrument, and which part of the preoperative image by CT or MRI corresponds to the part imaged by the endoscope. It is not a display.

  If the site being imaged by the endoscope (the surgical field displayed on the endoscope monitor) can be confirmed in the preoperative CT image or the like, for example, the operator can use the left hand. Confirm the place where the operation is to be performed directly with the endoscope held in the hand, and change any surgical instrument freely with the right hand while recognizing which part of the image is observed by CT etc. before surgery Surgical operation can be performed continuously. Thereby, a special marker is not required for the surgical instrument, and any instrument can be used without limitation.

As a technique for solving this, there is a technique for displaying a site observed by an endoscope on a preoperative image (CT / MRI) as disclosed in Patent Document 1.
JP 2001-204738 A

  However, the method described in Patent Document 1 is a distance measuring unit that measures the distance from the distal end of the insertion portion inserted into the patient's body to the surgical site in the patient's body (triangulation method using spot light irradiation or an ultrasonic sensor). Etc.) is used. Therefore, this method has a problem that a new special endoscope is required. In addition, when the structure of the endoscope is complicated by having the distance measuring means, the endoscope or these additional components may be damaged in daily work such as disinfection. In endoscopic surgery, a surgical instrument such as a suction tube often enters between the endoscope tip and the surgical site in the patient. In this case, the surgical instrument from the endoscope to the surgical site in the patient is used. It becomes difficult to measure the distance accurately. Furthermore, when an ultrasonic sensor is used, for example, in the cerebrospinal fluid or the exudate in the sinuses, the speed of the ultrasonic wave is different from that in the air, so it is necessary to know the speed accurately. However, body fluids, such as cerebrospinal fluid and exudate, vary greatly among individuals and are affected by the pathology of tumors, inflammation, etc., and their composition changes. Therefore, it is impossible to accurately calculate the ultrasonic velocity. is there. Therefore, there is a problem that it is impossible to accurately calculate the distance from the distal end of the endoscope to the surgical site in the patient.

  The present invention has been made to solve the above-described problems, and is a part that is imaged with an endoscope using a conventionally used endoscope without newly using a special endoscope. An object of the present invention is to provide a surgery support information display device, a surgery support information display method, and a surgery support information display program capable of accurately displaying which part of a patient corresponds to a patient.

  An operation support information display device according to the present invention includes an endoscope that is inserted into a patient and images the interior, and a surface that constitutes the interior of the patient into which the endoscope is inserted on the first coordinate axis. Patient shape acquisition means for acquiring first information indicating the first three-dimensional shape of the patient's surface, imaging means for imaging the patient's surface when an endoscope is inserted into the patient, and imaging means Obtained from the image of the surface of the patient imaged by the step 2 by the surface shape calculating means for calculating the second information indicating the second three-dimensional shape of the surface of the patient on the second coordinate axis, and the patient shape acquiring means. A plurality of reference points on one three-dimensional shape among the first three-dimensional shape indicated by the first information and the second three-dimensional shape indicated by the second information calculated by the surface shape calculating means For each of the plurality of reference points The corresponding point at the shortest distance from the reference point is calculated from the limited range of the other three-dimensional shape, and the first information and the second information are calculated based on the positional relationship between the plurality of reference points and the plurality of corresponding points. By matching the first coordinate axis with the second coordinate axis, and from the image picked up by the image pickup means, the second coordinate line matched with the first coordinate axis by the coordinate axis matching means. Endoscope optical axis calculating means for calculating a half line indicating the optical axis that is the imaging direction of the endoscope on the coordinate axes of the endoscope, and light that is the imaging direction of the endoscope calculated by the endoscope optical axis calculating means Intersection calculation means for calculating the intersection of the half line indicating the axis and the surface constituting the interior of the patient according to the first information acquired by the patient shape acquisition means, and information indicating the intersection calculated by the intersection calculation means Acquired by patient shape acquisition means And output means for superimposed output the information indicating the surface constituting the internal patient, characterized in that it comprises a.

  In the surgery support information display device according to the present invention, first information indicating the surface constituting the inside of the patient and the first three-dimensional shape of the surface of the patient is acquired by CT or the like. On the other hand, when the endoscope is inserted into the patient, the surface of the patient is imaged, and second information indicating the second three-dimensional shape of the surface of the patient is calculated from the image. Subsequently, a plurality of reference points are provided on one of the first three-dimensional shape and the second three-dimensional shape, and each of the plurality of reference points is at the shortest distance from the reference point. Corresponding points are calculated from a limited range on the other three-dimensional shape. Next, the first information and the second information are matched based on the positional relationship between the plurality of reference points and the plurality of corresponding points. And by this matching, the 1st coordinate axis concerning the 1st information acquired by CT etc. and the 2nd coordinate axis concerning the 2nd information computed from the picked-up image are in agreement.

  Note that when the first information and the second information are matched, it is assumed that the first three-dimensional shape and the second three-dimensional shape exist on a common coordinate axis. That is, the shortest distance is the distance between the reference point and the corresponding point when these two three-dimensional shapes exist on a common coordinate axis. The positional relationship between the reference point and the corresponding point is also defined on the assumption that the two three-dimensional shapes exist on a common coordinate axis.

  Subsequently, a half line indicating the optical axis that is the imaging direction of the endoscope is calculated from the captured image. An intersection point between the half line indicating the optical axis that is the imaging direction and the surface constituting the inside of the patient related to the first information acquired by CT or the like is calculated. This intersection point indicates the point (center point) at which the endoscope is imaging in the first information acquired by CT or the like. Information indicating the intersection is superimposed on the first information acquired by CT or the like and output.

  That is, as described above, in the operation support information display device according to the present invention, the first information indicating the first three-dimensional shape of the surface constituting the patient and the surface of the patient by CT or the like, and the patient from the outside It is possible to display which part of the patient corresponds to the part imaged by the endoscope using only the image obtained by imaging. Therefore, according to the surgery support information display apparatus according to the present invention, the above-described display can be performed without newly using a special endoscope. Further, according to the surgery support information display device according to the present invention, since the display is not affected by a liquid such as cerebrospinal fluid in the body of the patient, the display can be performed accurately. it can.

  In addition, in the surgery support information display device according to the present invention, when calculating the corresponding points on the second three-dimensional shape corresponding to the reference points on the first three-dimensional shape, the calculation range is limited. The For this reason, the corresponding points can be calculated earlier, and as a result, the first coordinate axis and the second coordinate axis can be matched in a shorter time.

  The surgery support information display device according to the present invention further includes a marker that is fixedly provided at a predetermined relative positional relationship with respect to the imaging direction of the endoscope. When the mirror is inserted, the marker is imaged together with the surface of the patient, and the endoscope optical axis calculation means is configured to match the first coordinate axis by the coordinate axis matching means from the image of the marker imaged by the imaging means. It is desirable to calculate the three-dimensional coordinates of the marker on the coordinate axis of the and calculate a half line indicating the optical axis that is the imaging direction of the endoscope from the positional relationship between the marker and the imaging direction of the endoscope .

  According to this configuration, when the endoscope is inserted into the patient, the coordinates of the marker fixed in a relative positional relationship with respect to the surface of the patient and the imaging direction of the endoscope are imaged. Subsequently, the three-dimensional coordinates of the marker are calculated from the captured image, and a half line indicating the optical axis that is the imaging direction of the endoscope is calculated from the calculated marker image. If the half line indicating the optical axis, which is the imaging direction of the endoscope, is calculated using the marker in this way, an accurate half line can be calculated more reliably. It can be carried out.

  In addition, since the operation support information display device according to the present invention only needs to acquire an image of the surface of the patient when the endoscope is inserted into the patient, it is not always necessary to include the endoscope as a component. . That is, the surgery support information display device according to the present invention includes a surface on the first coordinate axis that constitutes the interior of the patient into which the endoscope that is inserted into the patient and images the interior is inserted, and the patient's interior. Patient shape acquisition means for acquiring first information indicating the first three-dimensional shape of the surface, and surface image acquisition means for acquiring an image of the surface of the patient imaged when the endoscope is inserted into the patient; A surface shape calculating means for calculating second information indicating a second three-dimensional shape of the surface of the patient on the second coordinate axis from the image of the surface of the patient acquired by the surface image acquiring means, and a patient shape On one three-dimensional shape of the first three-dimensional shape indicated by the first information acquired by the acquisition means and the second three-dimensional shape indicated by the second information calculated by the surface shape calculation means A plurality of reference points are provided in the For each of the reference points, the corresponding point at the shortest distance from the reference point is calculated from the limited range of the other three-dimensional shape, and based on the positional relationship between the plurality of reference points and the corresponding points. By matching the first information and the second information, the first coordinate axis matching means for matching the first coordinate axis and the second coordinate axis, and the first obtained by the coordinate axis matching means from the image obtained by the surface image obtaining means. Endoscope optical axis calculation means for calculating a half line indicating the optical axis that is the imaging direction of the endoscope on the second coordinate axis that coincides with the coordinate axes of the endoscope, and the endoscope optical axis calculation means An intersection calculation means for calculating an intersection of a half line indicating an optical axis that is an imaging direction of the endoscope and a surface constituting the interior of the patient according to the first information acquired by the patient shape acquisition means; and an intersection calculation means Information indicating the calculated intersection And characterized in that it comprises and output means for outputting the superimposed information of a surface constituting the internal patient acquired by the patient shape acquiring unit.

  It is desirable that the intersection calculation means calculates the intersection between each surface constituting the polygon data and the half line indicating the optical axis that is the imaging direction of the endoscope, by using polygon data as a surface constituting the inside of the patient. . According to this configuration, the intersection can be easily and reliably calculated, and the present invention can be reliably implemented.

  The output means preferably outputs an image obtained by superimposing information indicating a location corresponding to the intersection on the image captured by the endoscope. According to this configuration, the operator or the like can simultaneously confirm both the content captured by the endoscope and the information on which part of the patient the portion captured by the endoscope corresponds to. Therefore, it is possible to provide more convenient surgical support.

  The coordinate axis coincidence means calculates the calculated corresponding point based on the position of the calculated corresponding point corresponding to the first reference point and the position of the second reference point different from the first reference point. It is desirable to calculate a center range and set the range as a limited range for calculating the corresponding point at the shortest distance from the second reference point. According to this configuration, an unknown corresponding point (corresponding point at the shortest distance from the second reference point) is calculated based on known information (the position of the known reference point and the position of the known corresponding point). Therefore, an unknown corresponding point can be calculated more quickly.

  The limited range is centered on the calculated corresponding point, the distance defined based on the position of the second reference point and the shortest distance field indicating a predetermined neighborhood space of the other three-dimensional shape, and the second Preferably, the radius is a value obtained by adding the distance between the reference point and the calculated corresponding point. According to this configuration, the distance defined based on the position of the calculated corresponding point, the position of the second reference point, and the shortest distance field indicating the predetermined neighborhood space of the other three-dimensional shape, Since the limited range for calculating the unknown corresponding point is set based on the distance between the reference point and the calculated corresponding point, the unknown corresponding point can be calculated earlier.

  The limited range is centered on the calculated corresponding point, the distance between the first reference point and the calculated corresponding point, the position of the second reference point, and a predetermined neighboring space of the other three-dimensional shape. It is preferable that the radius is a range obtained by adding a distance defined based on the shortest distance field shown and a distance between the first reference point and the second reference point. According to this configuration, the position of the calculated corresponding point, the distance between the first reference point and the calculated corresponding point, the position of the second reference point, and the predetermined neighboring space of the other three-dimensional shape A limited range for calculating an unknown corresponding point is set based on the distance defined based on the shortest distance field shown and the distance between the first reference point and the second reference point. Therefore, an unknown corresponding point can be calculated earlier.

  The coordinate axis matching means is present on the second three-dimensional shape indicated by the second information calculated by the surface shape calculating means, and the first information indicated by the first information acquired by the patient shape acquiring means. A corresponding point at the shortest distance from a reference point on the three-dimensional shape is calculated from a limited range of the second three-dimensional shape, and the first information is based on the positional relationship between the reference point and the corresponding point. It is desirable to match the first information and the second information by converting. According to this configuration, matching is performed by converting only the first information. That is, there is no need to process the second information. Therefore, the first coordinate axis and the second coordinate axis can be matched more easily.

  Incidentally, the present invention can be described as an invention of a surgery support information display apparatus as described above, and can also be described as an invention of a surgery support information display method and a surgery support information display program as follows. This is substantially the same invention only in different categories, and has the same operations and effects.

  That is, the operation support information display method according to the present invention includes a surface on the first coordinate axis that is inserted into a patient and into which an endoscope for imaging the interior is inserted and the patient's interior. A patient shape acquisition step for acquiring first information indicating the first three-dimensional shape of the surface; a surface image acquisition step for acquiring an image of the surface of the patient imaged when the endoscope is inserted into the patient; A surface shape calculation step for calculating second information indicating a second three-dimensional shape of the surface of the patient on the second coordinate axis from the image of the surface of the patient acquired in the surface image acquisition step; and a patient shape On one three-dimensional shape of the first three-dimensional shape indicated by the first information acquired in the acquisition step and the second three-dimensional shape indicated by the second information calculated in the surface shape calculation step A plurality of reference points are provided, and for each of the plurality of reference points, a corresponding point at the shortest distance from the reference point is calculated from a limited range of the other three-dimensional shape, and the plurality of reference points and the plurality of reference points are calculated. It is acquired in the coordinate axis matching step for matching the first coordinate axis and the second coordinate axis by matching the first information and the second information based on the positional relationship with the corresponding point, and the surface image acquisition step. An endoscope optical axis calculation step for calculating a half line indicating an optical axis that is an imaging direction of the endoscope on the second coordinate axis matched with the first coordinate axis in the coordinate axis matching step in the coordinate axis matching step; The intersection point of the half line indicating the optical axis that is the imaging direction of the endoscope calculated in the mirror optical axis calculation step and the surface constituting the interior of the patient related to the first information acquired in the patient shape acquisition step is calculated. An intersection calculating step, and an output step of superimposing and outputting information indicating the intersection calculated in the intersection calculating step on information indicating a surface constituting the inside of the patient acquired in the patient shape acquiring step. It is characterized by.

  An operation support information display program according to the present invention is provided on a first coordinate axis of a surface constituting an interior of a patient into which an endoscope that is inserted into the patient and images the interior is inserted, and the surface of the patient. A patient shape acquisition function for acquiring first information indicating the first three-dimensional shape, a surface image acquisition function for acquiring an image of the surface of the patient imaged when the endoscope is inserted into the patient, and a surface A surface shape calculation function for calculating second information indicating a second three-dimensional shape of the surface of the patient on the second coordinate axis from the image of the surface of the patient acquired by the image acquisition function; and a patient shape acquisition function A plurality of the first three-dimensional shape indicated by the first information acquired by the first information and the second three-dimensional shape indicated by the second information calculated by the surface shape calculation function on one three-dimensional shape. Multiple reference points For each of the reference points, the corresponding point at the shortest distance from the reference point is calculated from the limited range of the other three-dimensional shape, and based on the positional relationship between the plurality of reference points and the corresponding points. By matching the first information and the second information, the first coordinate axis matching function for matching the first coordinate axis and the second coordinate axis, and the first obtained by the coordinate axis matching function from the image obtained by the surface image obtaining function. An endoscope optical axis calculation function that calculates a half line indicating an optical axis that is an imaging direction of the endoscope on the second coordinate axis that is coincident with the coordinate axis of the endoscope, and an endoscope calculated by the endoscope optical axis calculation function An intersection calculation function for calculating an intersection between a half line indicating an optical axis that is an imaging direction of the endoscope and a surface constituting the interior of the patient related to the first information acquired by the patient shape acquisition function, and an intersection calculation function Information indicating the calculated intersection And characterized in that to execute an output function for superimposed output the information indicating the surface constituting the internal patient acquired by the patient shape acquiring function, to the computer.

  In the present invention, a part imaged by an endoscope using only information indicating the three-dimensional shape of the surface of the patient and the surface constituting the inside of the patient by CT and the like, and an image obtained by imaging the patient from the outside is used. It is possible to display which part of the patient corresponds. Therefore, according to the present invention, the above display can be performed without newly using a special endoscope. Further, according to the present invention, since the display is not affected by a liquid such as cerebrospinal fluid in the patient's body, the display can be performed accurately.

  Hereinafter, preferred embodiments of a surgery support information display device, a surgery support information display method, and a surgery support information display program according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. Further, the dimensional ratios in the drawings do not necessarily match those described.

  FIG. 1 is a configuration diagram schematically showing an embodiment of a surgery support information display device 1 according to the present invention. The surgery support information display device 1 is a device that provides information about an image captured by an endoscope to an operator or the like during surgery on a patient 60. Surgery in which the surgery support information display device 1 according to the present embodiment is used is intended for an operation in which imaging is performed by an endoscope, such as endoscopic surgery of a sinus in an otolaryngology department.

  As shown in FIG. 1, the surgery support information display device 1 includes an endoscope 11, a marker sphere 12, an imaging device 20, a CT device 30, a PC (Personal Computer) 40, and a monitor 50. It is configured.

  The endoscope 11 is a device that is operated by an operator and is inserted into a patient 60 to image the inside. The endoscope 11 has an elongated shape so that it can be inserted into a patient's living body, and a mechanism for imaging the inside of the patient 60 is provided at the distal end portion thereof. The mechanism is, for example, an imaging element such as a lens that is positioned so as to face the imaged portion and a CCD image sensor (Charge Coupled Device Image Sensor) that is provided at the determined position of the lens. The imaging direction A of the endoscope 11 is determined depending on how the mechanism is positioned. Usually, the optical axis direction of the lens is the imaging direction A of the endoscope 11. Information on the image captured by the endoscope 11 is output to the PC 40 connected to the endoscope 11 by a cable. The endoscope 11 does not need to have a special configuration, and a conventionally used endoscope can be used.

  The marker sphere 12 is a marker that is fixedly provided at a predetermined relative positional position with respect to the imaging direction of the endoscope 11. The marker sphere 12 is imaged by the imaging device 20, and the three-dimensional coordinates are obtained from the captured image. Specifically, the marker sphere 12 is a spherical member having a different size, which is fixed to the plurality of endoscopes 11 via a bar-shaped member 13. The reason why the sizes are different is to distinguish and detect each of the images captured by the imaging device 20.

  The position at which the marker ball 12 is provided in the endoscope 11 is a position that is not further inserted into the patient 60, further rearward from the portion inserted into the patient 60. Further, from the portion inserted into the patient 60 to the portion where the marker ball 12 is provided in the endoscope 11 so that the positional relationship between the marker ball 12 and the imaging direction A of the endoscope 11 is constant. It is made of a hard material and cannot be bent. However, since it is only necessary to know the positional relationship between the imaging direction of the endoscope 11 and the marker sphere 12, for example, only the distal end portion of the endoscope 11 may move in a predetermined direction. .

  Note that the marker provided in the endoscope 11 is in a relative positional relationship determined with respect to the imaging direction of the endoscope 11, and three-dimensional coordinates can be obtained from the image captured by the imaging device 20. Since it may be a thing, it does not necessarily need to be a spherical thing like the marker sphere 12 of this embodiment. Further, if the shape of the endoscope 11 itself can easily obtain a three-dimensional coordinate, the shape of the endoscope 11 itself becomes a marker, and therefore the marker sphere 12 is not necessarily provided.

  The imaging device 20 is an imaging unit that images the surface of the patient 60 and the marker sphere 12 when the endoscope 11 is inserted into the patient 60. As shown in FIG. 1, when the endoscope 11 is inserted from the nostril of the patient 60 and the head of the patient 60 is imaged by the endoscope 11, the face of the patient 60 and the marker ball 12 are The imaging device 20 is provided at a position where imaging can be performed. Specifically, for example, a CCD camera is used as the imaging device 20. The imaging device 20 is connected to the PC 40 and transmits information of the captured image to the PC 40.

  The image picked up by the image pickup device 20 is used to calculate the three-dimensional coordinates (three-dimensional position information) of the picked-up image. Therefore, it is necessary to provide the imaging apparatus 20 with a configuration necessary for this purpose. As a method for calculating a three-dimensional coordinate of an image captured from an image, there is a method using an optical method, and for example, a method described in Japanese Patent Laid-Open No. 2003-254732 can be used. When this method is used, a device for projecting a lattice pattern similar to white light similar to natural sunlight emitted from the xenon light is further provided in a range imaged by the imaging device 20 necessary for that purpose.

  According to the method described in Japanese Patent Laid-Open No. 2003-254732, imaging can be performed from a distance of 90 ± 10 cm in a measurement time of 1 second. The resolution is 0.1 to 0.6 mm. That is, a high-resolution color image having three-dimensional position information can be acquired in one second. Further, it is white light with an illuminance of about 28% during a cloudy daytime (outdoors), and three-dimensional position information can be acquired safely without using a laser or the like. FIG. 2 shows an example in which the face of the patient 60 is imaged by this method to obtain a high-resolution color image that has been processed and has three-dimensional position information.

  The CT apparatus 30 is patient shape acquisition means for acquiring first information indicating a surface constituting the inside of the patient 60 into which the endoscope 11 is inserted and a first three-dimensional shape of the surface of the patient 60. The surface constituting the patient 60 into which the endoscope 11 is inserted is, for example, the surface constituting the sinus when the endoscope 11 is inserted into the sinus cavity of the patient 60. In addition, the surface of the patient 60 is the face of the patient 60 in the above case. Here, the first information indicating the three-dimensional shape acquired by the CT apparatus 30 is configured by, for example, holding information indicating the three-dimensional shape for each coordinate with respect to a predetermined coordinate axis set in the CT apparatus 30 in advance. Has been. This coordinate axis is the first coordinate axis. That is, the first information is information on the first coordinate axis.

  The CT apparatus 30 scans an object using radiation or the like, and creates an image (CT image) of the internal structure processed by using a computer by cutting it at equal intervals (for example, 1 mm). It is configured as information indicating a three-dimensional shape, and an existing CT apparatus can be used. An example of a CT image is shown in FIG. The CT apparatus 30 is connected to the PC 40 and transmits information indicating the acquired three-dimensional shape of the patient 60 to the PC 40. Note that the CT apparatus 30 does not need to be installed at the same location as the imaging apparatus 20, and usually, imaging by the imaging apparatus 20 and acquisition of information indicating a three-dimensional shape by the CT apparatus 30 are performed separately. For example, a method described in Japanese Patent Laid-Open No. 2005-278992 can be used to construct information indicating a three-dimensional shape from a CT image.

  Note that the operation support information display device 1 only needs to be able to acquire information indicating a three-dimensional shape including the inside of the patient 60, and thus the CT device 30 is not necessarily used as the patient shape acquisition unit. For example, an MRI apparatus may be used.

  The PC 40 is an apparatus that receives information on an image captured by the imaging apparatus 20 and first information indicating the three-dimensional shape of the patient 60 acquired by the CT apparatus 30 and performs information processing on the information. is there. Specifically, the PC 40 is configured by hardware such as a CPU (Central Processing Unit) and a memory, and the following functions of the PC 40 are realized by operating these information processing apparatuses. As shown in FIG. 1, the PC 40 includes, as functional components, a patient shape acquisition unit 41, a captured image acquisition unit 42, a surface shape calculation unit 43, a coordinate axis matching unit 44, and an endoscope optical axis calculation. Unit 45, intersection calculation unit 46, and output unit 47.

  The patient shape acquisition unit 41 is means for receiving first information indicating the three-dimensional shape of the patient 60 transmitted from the CT apparatus 30. The patient shape acquisition unit 41 outputs the received first information to the coordinate axis matching unit 44, the intersection calculation unit 46, and the like as necessary. Note that the surgery support information display device 1 does not necessarily include the CT device 30 itself as a patient shape acquisition unit as in the present embodiment, and is included in the surgery shape information acquisition unit 41 (included in the surgery support information display device 1). It is only necessary to receive the first information indicating the three-dimensional shape of the patient 60 (captured by a non-CT apparatus).

  The captured image acquisition unit 42 is means for receiving information of an image captured and transmitted by the imaging device 20. That is, the captured image acquisition unit 42 is a surface image acquisition unit that acquires an image of the surface of the patient captured by the imaging device 20 when the endoscope 11 is inserted into the patient 60. The captured image acquisition unit 42 outputs the received image to the surface shape calculation unit 43, the endoscope optical axis calculation unit 45, and the like.

  The surface shape calculation unit 43 is a surface shape calculation unit that calculates second information indicating the second three-dimensional shape of the surface of the patient 60 from the image of the surface of the patient 60 captured by the imaging device 20. The surface of the patient 60 is the face of the patient 60 in this embodiment. As a method for calculating a three-dimensional shape from an image, for example, the method described in Japanese Patent Laid-Open No. 2003-254732 described above can be used. The second information is calculated, for example, by configuring information indicating a three-dimensional shape for each coordinate with respect to a predetermined coordinate axis set in the surface shape calculation unit 43 in advance. This coordinate axis is different from the first coordinate axis described above and is a second coordinate axis. That is, the second information is information on the second coordinate axis. The surface shape calculation unit 43 outputs the calculated second information to the coordinate axis matching unit 44.

  The coordinate axis matching unit 44 provides a plurality of reference points on the first three-dimensional shape indicated by the first information acquired by the patient shape acquisition unit 41, and for each of the plurality of reference points, from the reference point The corresponding points at the shortest distance are calculated from the limited range of the second three-dimensional shape indicated by the second information calculated by the surface shape calculating unit 43, and the plurality of reference points and the plurality of corresponding points are calculated. This is coordinate axis matching means for matching the first coordinate axis and the second coordinate axis by matching the first information and the second information based on the positional relationship between the first coordinate axis and the second information axis. That is, the coordinate axis matching unit 44 can process the first information acquired by the CT apparatus 30 and the second information calculated from the image captured by the imaging apparatus 20 on the same coordinate axis. Means. Note that formulas, rules, data, and the like necessary for calculating the limited range are stored in the coordinate axis matching unit 44 and are read out when calculating the limited range.

  The coordinate axis matching unit 44 uses an iterative nearest point algorithm (ICP algorithm) in order to match the first coordinate axis and the second coordinate axis. This algorithm is a technique for matching two figures composed of points.

  Processing of the coordinate axis matching unit 44 will be described with reference to FIG. In FIG. 4, for easy explanation, two-dimensional curves p and m are used as matching figures. In the example shown in FIG. 4, the curve m is fixed and the curve p is matched with the curve m. The matching in the two-dimensional space shown in FIG. 4 can be easily extended to the matching in the three-dimensional space as in this embodiment. For example, by replacing the curve p with the first three-dimensional shape indicated by the first information and replacing the curve m with the second three-dimensional shape indicated by the second information, Can be matched with the second information.

  The coordinate axis matching unit 44 first calculates the shortest distance field around the curve m. Here, the shortest distance field is a neighborhood space composed of points existing at a shortest distance (the length of the shortest path) from a certain figure. Next, the coordinate axis matching unit 44 selects an arbitrary reference point (first reference point) Pa on the curve p, and searches for a corresponding point Ma on the curve m that is the shortest distance from the reference point Pa ( calculate. The search for the corresponding point Ma is performed by calculating the distance from the reference point Pa for all points on the curve m and comparing the calculated distances. Note that the coordinate axis matching unit 44 can calculate in advance the position of a point on the curve m for calculating the distance from the reference point Pa using a predetermined relational expression, the arrangement of the points constituting the curve m, or the like. For example, the coordinate axis matching unit 44 can acquire the position of a point existing on the first three-dimensional shape or the second three-dimensional shape based on the first information or the second information.

  The positional relationship between the reference point and the corresponding point is defined on the assumption that the two three-dimensional shapes exist on a common coordinate axis. For example, the calculation of the distance between the reference point Pa and a point on the curve m assumes that the curve p and the curve m exist on a common coordinate axis. That is, the shortest distance is the distance between the reference point and the corresponding point when these two curves exist on a common coordinate axis.

  Next, the coordinate axis matching unit 44 selects another reference point (second reference point) Pb on the curve p, and searches for the corresponding point Mb on the curve m that is the shortest distance from the reference point Pb ( calculate. At this time, the coordinate axis matching unit 44 calculates distances between a plurality of points existing in a limited range on the curve m and the reference point Pa.

  For this purpose, the coordinate axis coincidence unit 44 determines the coordinate values of the reference points Pa and Pb, the coordinate value of the corresponding point Ma, the distance 1 between the reference point Pa and the corresponding point Ma, and the reference point Pb and the corresponding point Ma. The distance d, the coordinate value of the reference point Pb, and the distance l ′ determined by the shortest distance field around the curve m are acquired. These values are known or can be calculated from known values. Then, the coordinate axis matching unit 44 determines a range for searching for the corresponding point Mb based on these acquired values.

In this case, the following triangular inequality is established from the positional relationship among the reference point Pb, the known corresponding point Ma, and the unknown corresponding point Mb. Incidentally, d _m is the distance between the known corresponding points Ma and unknown correspondence point Mb.
d _m ≦ l ′ + d

Therefore, the range of the curve m to be searched for the unknown corresponding point Mb is limited to the circle C whose center is the known corresponding point Ma and whose radius is expressed by the following equation.
r = l ′ + d (1)
The radius r may be calculated using the following formula. Here, d _p is a distance between known reference points Pa and Pb.
r = l + l ′ + d _p (2)

  In addition, since the figure shown by the 1st and 2nd information is a three-dimensional shape, in this embodiment, the coordinate axis matching part 44 is a corresponding point within the sphere which has the predetermined radius calculated according to the said theory. Explore.

  That is, the coordinate axis coincidence unit 44 calculates the distance between the point and the reference point Pb only for a point in the range of the radius r defined by the above formula (1) or (2) from the corresponding point Ma in the curve m. calculate. Then, the coordinate axis matching unit 44 determines the point having the shortest distance from the reference point Pb as the corresponding point Mb. The coordinate axis matching unit 44 can calculate and acquire the position of a point on the curve m in advance. Therefore, the coordinate axis matching unit 44 determines the reference point Pb based on the position of the point on the curve m acquired in advance, the position of the corresponding point Ma, and the radius r defined by the above formula (1) or (2). It is possible to obtain a point on the curve m for which the distance to be calculated.

  Thereafter, the coordinate axis matching unit 44 searches (calculates) a corresponding point for one or more other reference points on the curve p. The method for calculating the corresponding point is the same as the method for calculating the corresponding point Mb. That is, when calculating the second and subsequent corresponding points, the coordinate axis matching unit 44 calculates the corresponding points from a limited range on the curve m.

Subsequently, the coordinate axis matching unit 44 calculates the average value of the distance between the reference point and the corresponding point corresponding to the reference point. Subsequently, the coordinate axis matching unit 44 calculates the rotation matrix R and the translation matrix T based on the positional relationship between the plurality of calculated reference points and corresponding points, and performs rigid body transformation using these two matrices. The rigid body transformation is expressed by the following equation, where p _i-1 is the position of the curve p before the rigid body transformation, and p _i is the position of the curve p after the rigid body transformation.
p _i = R * p _i-1 + T

  By this rigid transformation, the curve p rotates or translates. The coordinate axis coincidence unit 44 repeatedly executes the processing from the calculation of the corresponding point to the rigid transformation until the difference between the average value calculated this time and the average value calculated last time becomes less than a predetermined threshold value.

  By such processing of the coordinate axis coincidence unit 44, a function for converting one of the first coordinate axis and the second coordinate axis to the other coordinate axis is calculated. The coordinate axis matching unit 44 outputs a function or the like for converting the coordinate axis calculated as a result of matching the coordinate axes to the endoscope optical axis calculation unit 45, the intersection calculation unit 46, the output unit 47, and the like as necessary. After the process of matching the coordinate axes, the above functions and the like are applied to the information indicating the three-dimensional shape in the endoscope optical axis calculation unit 45, the intersection calculation unit 46, the output unit 47, and the like. The information indicating the three-dimensional shape by 30 and the information indicating the three-dimensional shape calculated from the image captured by the imaging device 20 are processed on the same coordinate axis.

  The endoscope optical axis calculation unit 45 calculates the three-dimensional coordinates of the marker sphere 12 on the coordinate axis matched by the coordinate axis matching unit 44 from the image of the marker sphere 12 imaged by the imaging device 20, and the marker sphere 12 12 is an endoscope optical axis calculation unit that calculates a half line indicating the optical axis that is the imaging direction A of the endoscope 11 from the positional relationship between the imaging direction 12 of the endoscope 11 and the imaging direction A of the endoscope 11. In addition, the position which becomes the starting point of a half line is included in the half line which shows the optical axis which is the imaging direction A said here. That is, the half line indicating the optical axis that is the imaging direction A of the endoscope 11 indicates from which point the imaging is performed. The endoscope optical axis calculation unit 45 holds information indicating the positional relationship between the marker sphere 12 and the imaging direction A of the endoscope 11 in advance. As the information indicating the positional relationship, specifically, for example, a predetermined position of the base of the endoscope 11 so as to pass through the optical axis of the endoscope 11 (a half line indicating the optical axis that is the imaging direction A). Information indicating the position of the tip and the positional relationship between the two points and the marker sphere 12 is used.

  The calculation of the half line indicating the optical axis that is the imaging direction A of the endoscope 11 is specifically performed as follows. First, the endoscope optical axis calculation unit 45 calculates the three-dimensional shape and the three-dimensional coordinates from the marker sphere 12 imaged by the imaging device 20 in the same manner as the method of calculating the three-dimensional shape in the surface shape calculation unit 43. calculate. When a plurality of marker spheres 12 are provided, the endoscope optical axis calculation unit 45 identifies each marker sphere 12 based on the size of the marker sphere 12, for example.

  Subsequently, the endoscope optical axis calculation unit 45 includes the calculated three-dimensional coordinates of the marker sphere 12 on the coordinate axis and information indicating the positional relationship between the marker sphere 12 and the imaging direction A of the endoscope 11. Then, a half line indicating the optical axis that is the imaging direction A on the coordinate axis is calculated. The endoscope optical axis calculation unit 45 outputs half-line information indicating the optical axis that is the calculated imaging direction to the intersection calculation unit 46.

  The intersection calculation unit 46 relates to the half line indicating the optical axis that is the imaging direction A of the endoscope 11 calculated by the endoscope optical axis calculation unit 45 and the first information acquired by the patient shape acquisition unit 41. It is an intersection calculation means for calculating an intersection with a surface constituting the inside of the patient 60. This intersection point indicates a point (center point) where the endoscope 11 is imaging in the first information indicating the three-dimensional shape by the CT apparatus 30. Specifically, the intersection calculation unit 46 converts the surfaces constituting the inside of the patient 60 into polygon data, and shows each surface constituting the polygon data and the optical axis that is the imaging direction A of the endoscope 11. The intersection point with is calculated. The calculation of the intersection will be described in detail later. The intersection calculation unit 46 outputs the calculated intersection information to the output unit 47.

  The output unit 47 superimposes the information indicating the intersection calculated by the intersection calculation unit 46 on the CT image which is information indicating the surface constituting the inside of the patient 60 acquired by the patient shape acquisition unit 41, and monitors the monitor 50. Is an output means for outputting to. The output unit 47 may output the endoscopic image captured by the endoscope 11 and input to the PC 40 to the monitor 50 together.

  The monitor 50 displays information input from the PC 40. By referring to the monitor 50, the surgeon can know which part of the patient 60 is imaging the endoscope 11. The above is the configuration of the surgery support information display device 1.

  Subsequently, the operation (surgery support information display method) of the surgery support information display device 1 will be described with reference to the flowchart of FIG. This operation is, for example, an operation when performing treatment or the like by inserting the endoscope 11 during surgery on the patient 60. In this description, the processing before the operation and the processing at the time of the operation will be described separately.

  First, before the operation, CT scan imaging is performed on the patient 60 using the CT apparatus 30 (S01, patient shape acquisition step). This CT scan imaging is performed on the part of the patient 60 into which the endoscope 11 is inserted. Thereby, the 1st information which shows the 1st three-dimensional shape of the face which is the surface of patient 60, and the field which constitutes the inside of patient 60 into which endoscope 11 is inserted is acquired. The first information acquired by CT scan imaging performed by the CT apparatus 30 is transmitted to the PC 40. In the PC 40, the information is acquired by the patient shape acquisition unit 41 and stored in the PC 40 (S02, patient shape acquisition step). The above is processing before surgery, for example, performed on the day before surgery.

  Subsequently, processing during surgery will be described. First, the patient 60 enters the operating room, and as shown in FIG. 1, the endoscope 11 is placed on the back on the operating table 70 so that the endoscope 11 can be inserted through the nostril. After the patient 60 is placed and before the endoscope 11 is inserted, the placed patient 60 is imaged by the imaging device 20 (S03, surface image acquisition step). The captured image is transmitted from the imaging device 20 to the PC 40 and received by the captured image acquisition unit 42 in the PC 40. The received image is output from the captured image acquisition unit 42 to the surface shape calculation unit 43.

  The surface shape calculation unit 43 calculates second information indicating the second three-dimensional shape of the face, which is the surface of the patient 60, from the image (S04, surface shape calculation step). The calculated second information is output from the surface shape calculation unit 43 to the coordinate axis matching unit 44. Information indicating the three-dimensional shape of the patient 60 by the CT apparatus 30 stored in the PC 40 is output from the patient shape obtaining unit 41 to the coordinate axis matching unit 44 at the same timing.

  The coordinate axes related to the first information by the CT apparatus 30 and the second information calculated from the image by the imaging apparatus 20 do not match. When these pieces of information are displayed together, as shown in FIG. 6A, the information 81 by the CT apparatus 30 and the information 82 by the image of the imaging apparatus 20 are not aligned.

  Here, as shown in FIG. 6B, the coordinate shape matching unit 44 matches the shape of the face in the two pieces of information 81 and 82 to match the coordinate axes related to the two pieces of information (S05, coordinate axis match). Step). Note that the matching parts are set in advance, such as the whole face or a characteristic part such as the nose or cheek of the face.

  The process of matching the coordinate axes will be described in detail with reference to the flowchart of FIG. First, the 1st information acquired by patient shape acquisition part 41 and the 2nd information calculated by surface shape calculation part 43 are prepared (Step S51). Subsequently, a corresponding point corresponding to the reference point on the first three-dimensional shape indicated by the first information is extracted from the second three-dimensional shape indicated by the second information (step S52). This extraction is repeated until all the corresponding points for a predetermined number are extracted (step S53; NO). At this time, in the extraction of the second and subsequent corresponding points, the corresponding points are extracted from a limited range of the second three-dimensional shape.

  When all the corresponding points are extracted (step S53; YES), the average value of the distance between the reference point and the corresponding point is calculated (step S54). Subsequently, a rotation matrix and a translation matrix are calculated based on the positional relationship between the plurality of calculated reference points and corresponding points (step S55). Subsequently, the first three-dimensional shape indicated by the first information is rigidly transformed based on the rotation matrix and the translation matrix (step S56). Subsequently, it is determined whether or not the difference between the average value calculated this time and the average calculated last time is less than a threshold value (step S57).

  Here, if the difference between the average values is equal to or greater than the threshold value (step S57; NO), the process after step S52 is repeatedly executed. If the difference is less than the threshold value (step S57; YES), the process ends. To do. As a result, the first coordinate axis and the second coordinate axis are matched.

  The information related to the coordinate axes that coincided is output from the coordinate axis coincidence unit 44 to the endoscope optical axis calculation unit 45, the intersection calculation unit 46, and the output unit 47, respectively, and the coordinate axes are converted. Information processing on the three-dimensional shape is performed with reference to the coordinate axes. The above is the process until the start of surgery.

  Subsequently, an operation is started, and the endoscope 11 is inserted into the patient 60 by the operator. At this time, the head of the patient 60 is prevented from moving after the processing of S03 to S05 is performed. This is to prevent the coordinate axes from shifting. When the endoscope 11 is inserted into the patient 60, the patient 60 and the marker sphere 12 are imaged by the imaging device 20 (S06, surface image acquisition step). As shown in FIG. 8A, the captured image includes a marker sphere 12 (image thereof). The captured image is transmitted from the imaging device 20 to the PC 40 and received by the captured image acquisition unit 42 in the PC 40. The received image is output from the captured image acquisition unit 42 to the endoscope optical axis calculation unit 45.

  Subsequently, as shown in FIG. 8B, the three-dimensional coordinates of the marker sphere 12 are calculated from the image by the endoscope optical axis calculation unit 45 (S07, endoscope optical axis calculation step). Subsequently, information indicating the positional relationship between the marker sphere 12 and the imaging direction A of the endoscope 11 held in advance is calculated from the three-dimensional coordinates of the marker sphere 12 calculated by the endoscope optical axis calculator 45. Based on this, a half line indicating the optical axis that is the imaging direction A of the endoscope 11 on the coordinate axes that are matched is calculated (S08, endoscope optical axis calculating step).

  The calculated half-line information is output from the endoscope optical axis calculation unit 45 to the intersection calculation unit 46. At the same timing, the first information by the CT apparatus 30 stored in the PC 40 is output from the patient shape acquisition unit 41 to the intersection calculation unit 46. Subsequently, the intersection calculation unit 46 calculates the intersection between the half line indicating the optical axis that is the imaging direction A of the endoscope 11 and the surface constituting the inside of the patient 60 (S09, intersection calculation step).

  The intersection is calculated as follows. First, the intersection calculation unit 46 converts the first information from the CT apparatus 30 into polygon data. By this conversion, the surface constituting the patient 60 (inside) is constituted by, for example, a large number of triangles. Next, the intersection of each triangle and the half line indicating the optical axis that is the imaging direction A of the endoscope 11 is calculated.

The calculation of the intersection will be described with reference to FIG. A triangle can be expressed by the following expression, where PT is a reference point of a triangle constituting a polygon, two vectors of the triangle are vecA and vecB, and two parameters are α and β.
P1 = PT + αvecA + βvecB
Further, assuming that the reference point of the imaging direction A of the endoscope 11 (for example, the point at the tip of the endoscope 11) is PL, the vector indicating the direction of the half line is vecC, and the parameter is γ, The imaging direction A of the mirror 11 can be expressed by the following equation.
P2 = PL + γvecC

Here, when the two intersect, P1 = P2. The condition that the point where P1 = P2 exists inside the triangle is when the parameter satisfies the following condition.
Condition 1: 0 <α, 0 <β
Condition 2: 0 <α + β <1
Condition 3: γ> 0
Points that satisfy these conditions are derived for the triangles that make up all the polygon data, and the distance between all these points and the point at the tip of the endoscope 11 is calculated. The intersection at which this distance is the smallest is the intersection of the half line indicating the optical axis that is the imaging direction A of the endoscope 11 and the surface that forms the inside of the patient 60.

  Information on the coordinates of the calculated intersection is output from the intersection calculation unit 46 to the output unit 47. At this timing, a CT image that is information indicating a surface constituting the inside of the patient 60 is output from the patient shape acquisition unit 41 to the output unit 47. The information on the intersection is input to the monitor 50 by the output unit 47 by superimposing the CT image, which is information indicating the surface constituting the inside of the patient 60, at a location corresponding to the coordinates of the intersection. The input image is displayed on the monitor 50 (S10, output step). Specifically, for example, as shown in FIG. 10, the intersection point is displayed as a cross display 90 so that the point where the part imaged by the endoscope 11 is located can be seen. By referring to the displayed image, the surgeon can know which part of the patient 60 is imaging the endoscope 11.

  In addition, the image captured by the endoscope 11 is also received by the PC 40 and output from the output unit 47 to the monitor 50, so that the position where the image is captured by the endoscope 11 is located at any point. It is desirable to be displayed together with the display of whether or not For example, as in a screen example 91 shown in FIG. 11, an image 92 captured by the endoscope 11 and a CT image 93 a that indicates where the position captured by the endoscope 11 is located. ~ 93c should be shown on one screen. In addition, it is desirable to superimpose information (for example, a cross (+) mark) 94 indicating a location corresponding to the intersection on the image of the endoscope 11 at that time. The position of the information indicating the location corresponding to the intersection in the image of the endoscope 11 is stored in advance in the output unit 47 according to the imaging direction of the endoscope 11 such as the center of the image, for example. In order to make it easier for the surgeon to understand the corresponding location, the CT images 93a to 93c preferably display a plurality of different cross sections (preferably cross sections that are orthogonal to each other). In FIG. 11, the CT image 93a has a transverse cross section, the CT image 93b has a coronal cross section, and the CT image 93c has a sagittal cross section. Further, in the CT images 93 a to 93 c, in addition to the intersection information (indicated by a square 95 of a different color from the image), the direction in which the endoscope 11 is inserted (light that is the imaging direction of the endoscope 11). It is desirable to show axis 96.

  The processes from S06 to S10 are repeatedly performed at equal intervals such as 1 second. In addition, although the process of S03-S05 and the process of S06-S10 in PC40 are different, for example, if the process of matching the coordinate axes of S05 is performed, the process may be automatically shifted to the process after S06. Good. Further, the process may be switched by an operation of an operator or the like.

  Further, even when the head of the patient 60 is moved after the initial alignment by the processing of S03 to S05, the processing of S03 to S05 may be performed again to perform alignment again. The re-alignment may be performed, for example, by an operation of an operator or the like, or the image at the time of alignment is compared with the subsequent image to detect that the head has moved, and It may be performed as a trigger.

  As described above, in the surgery support information display device 1 according to the present embodiment, the first information indicating the surface constituting the inside of the patient 60 by the CT device 30 and the first three-dimensional shape of the surface of the patient, and imaging It is possible to display which part of the patient 60 corresponds to the part imaged by the endoscope 11 using only the image obtained by imaging the patient 60 from the outside by the apparatus 20. Therefore, according to the surgery support information display device 1 according to the present embodiment, the above-described display can be performed without newly using a special endoscope. In addition, according to the surgery support information display device 1 according to the present embodiment, the display is not affected by a liquid such as cerebrospinal fluid in the patient's body, and thus the above display is performed accurately. be able to. Therefore, if the surgery support information display device 1 according to the present embodiment is used, the surgery can be performed safely and accurately.

  Further, in the surgery support information display device 1 according to the present embodiment, when calculating the corresponding point on the second three-dimensional shape corresponding to the reference point on the first three-dimensional shape, the calculation range is Limited. For this reason, the corresponding points can be calculated earlier, and as a result, the first coordinate axis and the second coordinate axis can be matched in a shorter time.

  In addition, it is not necessary to mark the patient 60 during CT scan imaging (S01), and it is convenient because CT scan imaging can be performed as usual. Further, it is not necessary to fix the patient 60 with pins or the like for alignment or the like. Even if the patient 60 is moved during the operation, the alignment can be easily performed. Each component of the surgery support information display device 1 is relatively inexpensive and can be realized at low cost.

  Moreover, if the marker sphere 12 which is a marker is provided in the endoscope 11 as in this embodiment, and a half line indicating the optical axis which is the imaging direction of the endoscope 11 is calculated using the marker. Since an accurate half line can be calculated more reliably, the above-mentioned display can be performed more reliably. However, a half line indicating the optical axis that is the imaging direction of the endoscope 11 may be calculated without using the marker. For example, in the present embodiment, similar to the method of calculating the surface of the patient from the image by the imaging device 20, the shape of the endoscope 11 is calculated, and the optical axis that is the imaging direction of the endoscope 11 is calculated from the shape. The half line shown may be calculated.

  Note that the present invention can be implemented as long as an image or information is input to the PC 40. Therefore, the surgery support information display device 1 does not necessarily include hardware such as the endoscope 11, the imaging device 20, and the CT device 30. It is not necessary to have. That is, the present invention can be implemented as long as each component 41 to 47 included in the PC 40 is provided.

  Further, as in the present embodiment, if the three-dimensional information of the patient 60 by the CT apparatus 30 is used as polygon data and the intersection point is obtained, the intersection point can be calculated reliably, and the present invention can be reliably implemented. it can.

  In addition, if the image captured by the endoscope 11 is also displayed together with information on which part of the patient 60 the portion captured by the endoscope 11 corresponds to, the operator or the like However, since it is possible to simultaneously confirm both the content captured by the endoscope 11 and the information on which part of the patient 60 the portion captured by the endoscope 11 corresponds to, it is more convenient. High surgical support can be performed.

  Further, for example, as illustrated in FIG. 4, the coordinate axis matching unit 44 has a radius centered on the corresponding point Ma based on the position of the calculated corresponding point Ma corresponding to the reference point Pa and the position of the reference point Pb. The circle C of r is calculated, and the circle C is set as a limited range of the curve m. Thereby, since the limited range for calculating the unknown corresponding point Mb is set, the corresponding point Mb can be calculated more quickly. The same applies to the calculation of other corresponding points.

Further, as shown in FIG. 4, for example, the coordinate axis matching unit 44 is based on the calculated corresponding point Ma, the position of the corresponding point Ma, the position of the reference point Pb, and the shortest distance field of the curve m. A circle C having a radius r of a value (l ′ + d) calculated in this way is defined as a limited range. In addition, for example, as shown in FIG. 4, the coordinate axis matching unit 44 is centered on the calculated corresponding point Ma, the position of the corresponding point Ma, the position of the reference point Pa, the position of the reference point Pb, and a curve A circle C having a radius r of a value (l + l ′ + d _p ) calculated based on the shortest distance field of m may be a limited range. Thereby, since the limited range (circle C) for calculating the unknown corresponding point Mb is set, the corresponding point Mb can be calculated earlier. The same applies to the calculation of other corresponding points.

  The coordinate axis matching unit 44 is the first information that exists on the second three-dimensional shape indicated by the second information calculated by the surface shape calculation unit 43 and is acquired by the patient shape acquisition unit 41. The corresponding point at the shortest distance from the reference point on the first three-dimensional shape shown is calculated from the limited range of the second three-dimensional shape, and based on the positional relationship between the reference point and the corresponding point By converting the first information, the first information and the second information are matched. Thus, matching is performed by converting only the first information acquired by the CT apparatus 30. That is, it is not necessary to process the second information calculated based on the image of the imaging device 20. Therefore, the first coordinate axis and the second coordinate axis can be matched more easily.

  Subsequently, a description will be given of a surgery support information display program for causing a computer to execute the processing for displaying the series of surgery support information described above. As shown in FIG. 12, the surgery support information display program 82 is stored in a program storage area 80a formed in a recording medium 80 provided in the computer.

  The surgery support information display program 81 includes a main module 81a that centrally controls display processing of surgery support information, a patient shape acquisition module 81b, a captured image acquisition module 81c, a surface shape calculation module 81d, and a coordinate axis matching module 81e. An endoscope optical axis calculation module 81f, an intersection calculation module 81g, and an output module 81h. Functions realized by executing the patient shape acquisition module 81b, the captured image acquisition module 81c, the surface shape calculation module 81d, the coordinate axis matching module 81e, the endoscope optical axis calculation module 81f, the intersection calculation module 81g, and the output module 81h The functions of the patient shape acquisition unit 41, the captured image acquisition unit 42, the surface shape calculation unit 43, the coordinate axis matching unit 44, the endoscope optical axis calculation unit 45, the intersection calculation unit 46, and the output unit 47 of the PC 40 described above, respectively. It is.

  The surgery support information display program 81 may be partly or wholly transmitted via a transmission medium such as a communication line, received by another device, and recorded (including installation).

It is a figure which shows the structure of the surgery assistance information display apparatus which concerns on embodiment of this invention. It is a figure which shows the image of a patient's face which was imaged with the imaging device, was information-processed, and had 3D position information. It is a figure which shows CT image acquired by CT apparatus. It is a figure which shows the concept of a process of the coordinate axis matching part shown in FIG. It is a flowchart which shows the process in the surgery assistance information display apparatus which concerns on embodiment of this invention. It is a figure which shows the matching process of the information which shows the patient's three-dimensional shape by CT apparatus, and the information which shows the patient's three-dimensional shape calculated from the image by an imaging device. It is a flowchart which shows the coordinate axis matching process shown in FIG. It is the figure which showed the process which calculates the three-dimensional coordinate of a marker ball from the image by an imaging device. It is the figure which showed the intersection of the triangle which comprises a patient's surface, and the half line which shows the optical axis which is the imaging direction of an endoscope. It is a figure which shows the CT image in which the point imaged with the endoscope was shown. It is a figure which shows the CT image in which the point imaged with the endoscope was shown, and the image imaged with the endoscope. It is a figure which shows the structure of the surgery assistance information display program which concerns on this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Surgery support information display apparatus, 11 ... Endoscope, 12 ... Marking ball, 20 ... Imaging apparatus, 30 ... CT apparatus, 40 ... PC, 41 ... Patient shape acquisition part, 42 ... Captured image acquisition part, 43 ... Surface Shape calculation unit 44 ... Coordinate axis matching unit 45 ... Endoscope optical axis calculation unit 46 ... Intersection calculation unit 47 ... Output unit 50 ... Monitor 60 ... Patient 70 ... Operating table 80 ... Recording medium 80a ... Program storage area, 81 ... Surgery support information display program, 81a ... Main module, 81b ... Patient shape acquisition module, 81c ... Captured image acquisition module, 81d ... Surface shape calculation module, 81e ... Coordinate axis matching module, 81f ... Endoscope Optical axis calculation module, 81g ... intersection calculation module, 81h ... output module.

Claims (11)

An endoscope that is inserted into the patient and images the interior;
Patient shape acquisition means for acquiring first information indicating a surface constituting the inside of the patient into which the endoscope is inserted and a first three-dimensional shape of the surface of the patient on the first coordinate axis; ,
Imaging means for imaging the surface of the patient when the endoscope is inserted into the patient;
Surface shape calculating means for calculating second information indicating a second three-dimensional shape of the surface of the patient on a second coordinate axis from the image of the surface of the patient imaged by the imaging means;
Of the first three-dimensional shape indicated by the first information acquired by the patient shape acquisition means and the second three-dimensional shape indicated by the second information calculated by the surface shape calculation means A plurality of reference points are provided on one three-dimensional shape, and for each of the plurality of reference points, the corresponding point at the shortest distance from the reference point is calculated from the limited range of the other three-dimensional shape, A coordinate axis that matches the first coordinate axis and the second coordinate axis by matching the first information and the second information based on the positional relationship between the reference point and the corresponding points. Matching means;
A half line indicating an optical axis that is an imaging direction of the endoscope on the second coordinate axis matched with the first coordinate axis by the coordinate axis matching means is calculated from an image captured by the imaging means. Endoscope optical axis calculation means;
The inside of the patient according to the first information acquired by the patient shape acquisition unit and the half line indicating the optical axis that is the imaging direction of the endoscope calculated by the endoscope optical axis calculation unit is configured. An intersection calculation means for calculating an intersection with the surface;
Output means for superimposing and outputting the information indicating the intersection calculated by the intersection calculation means on the information indicating the surface constituting the inside of the patient acquired by the patient shape acquisition means;
An operation support information display device comprising: A marker that is fixed and provided at a predetermined relative positional relationship with respect to the imaging direction of the endoscope;
The imaging means images the marker together with the surface of the patient when the endoscope is inserted into the patient;
The endoscope optical axis calculation means is a three-dimensional image of the marker on the second coordinate axis that is matched with the first coordinate axis by the coordinate axis matching means from the image of the marker imaged by the imaging means. Calculate coordinates, and calculate a half line indicating the optical axis that is the imaging direction of the endoscope from the positional relationship between the marker and the imaging direction of the endoscope.
The surgery support information display device according to claim 1, wherein On the first coordinate axis, a first surface showing a surface constituting the inside of a patient into which an endoscope that is inserted into the patient and images the inside is inserted, and a first three-dimensional shape of the surface of the patient are shown. Patient shape acquisition means for acquiring the information of
Surface image acquisition means for acquiring an image of the surface of the patient imaged when the endoscope is inserted into the patient;
Surface shape calculating means for calculating second information indicating a second three-dimensional shape of the surface of the patient on a second coordinate axis from the image of the surface of the patient acquired by the surface image acquiring means;
Of the first three-dimensional shape indicated by the first information acquired by the patient shape acquisition means and the second three-dimensional shape indicated by the second information calculated by the surface shape calculation means A plurality of reference points are provided on one three-dimensional shape, and for each of the plurality of reference points, the corresponding point at the shortest distance from the reference point is calculated from the limited range of the other three-dimensional shape, A coordinate axis that matches the first coordinate axis and the second coordinate axis by matching the first information and the second information based on the positional relationship between the reference point and the corresponding points. Matching means;
From the image acquired by the surface image acquisition means, a half line indicating the optical axis that is the imaging direction of the endoscope on the second coordinate axis matched with the first coordinate axis by the coordinate axis matching means is calculated. Endoscope optical axis calculating means for
The inside of the patient according to the first information acquired by the patient shape acquisition unit and the half line indicating the optical axis that is the imaging direction of the endoscope calculated by the endoscope optical axis calculation unit is configured. An intersection calculation means for calculating an intersection with the surface;
Output means for superimposing and outputting the information indicating the intersection calculated by the intersection calculation means on the information indicating the surface constituting the inside of the patient acquired by the patient shape acquisition means;
An operation support information display device comprising:   The intersection calculation means calculates the intersection between each surface forming the polygon data and a half line indicating the optical axis that is the imaging direction of the endoscope, using polygon data as a surface forming the interior of the patient. The surgery support information display device according to any one of claims 1 to 3, wherein   5. The output device according to claim 1, wherein the output unit outputs an image obtained by superimposing information indicating a location corresponding to the intersection on an image captured by the endoscope. 6. The operation support information display device described.   The coordinate axis matching means is configured to calculate the calculated reference point based on the calculated position of the corresponding point corresponding to the first reference point and the position of the second reference point different from the first reference point. 2. A range centering on the corresponding point is calculated, and the range is the limited range for calculating the corresponding point at the shortest distance from the second reference point. The surgery assistance information display apparatus as described in any one of -5.   The limited range is a distance defined on the basis of the position of the second reference point and the shortest distance field indicating a predetermined neighboring space of the other three-dimensional shape with the calculated corresponding point as the center. The surgery support information display device according to claim 6, wherein a radius is a value obtained by adding a distance between the second reference point and the calculated corresponding point.   The limited range is centered on the calculated corresponding point, the distance between the first reference point and the calculated corresponding point, the position of the second reference point, and the other three-dimensional shape. A radius defined by adding a distance defined based on a shortest distance field indicating a predetermined neighborhood space and a distance between the first reference point and the second reference point. The operation support information display device according to claim 6.   The coordinate axis matching means is present on the second three-dimensional shape indicated by the second information calculated by the surface shape calculating means, and indicated by the first information acquired by the patient shape acquiring means. The corresponding point at the shortest distance from the reference point on the first three-dimensional shape is calculated from the limited range of the second three-dimensional shape, and based on the positional relationship between the reference point and the corresponding point The surgery support information according to any one of claims 1 to 8, wherein the first information and the second information are matched by converting only the first information. Display device. On the first coordinate axis, a first surface showing a surface constituting the inside of a patient into which an endoscope that is inserted into the patient and images the inside is inserted, and a first three-dimensional shape of the surface of the patient are shown. A patient shape acquisition step for acquiring the information of
A surface image acquisition step of acquiring an image of the surface of the patient imaged when the endoscope is inserted into the patient;
A surface shape calculating step of calculating second information indicating a second three-dimensional shape of the surface of the patient on a second coordinate axis from the image of the surface of the patient acquired in the surface image acquiring step;
Of the first three-dimensional shape indicated by the first information acquired in the patient shape acquisition step and the second three-dimensional shape indicated by the second information calculated in the surface shape calculation step A plurality of reference points are provided on one three-dimensional shape, and for each of the plurality of reference points, the corresponding point at the shortest distance from the reference point is calculated from the limited range of the other three-dimensional shape, A coordinate axis that matches the first coordinate axis and the second coordinate axis by matching the first information and the second information based on the positional relationship between the reference point and the corresponding points. A matching step,
From the image acquired in the surface image acquisition step, a half line indicating the optical axis that is the imaging direction of the endoscope on the second coordinate axis matched with the first coordinate axis in the coordinate axis matching step is calculated. An endoscope optical axis calculating step,
The inside of the patient according to the first information acquired in the patient shape acquisition step and the half line indicating the optical axis that is the imaging direction of the endoscope calculated in the endoscope optical axis calculation step is configured. An intersection calculation step for calculating an intersection with the surface;
An output step of superimposing and outputting information indicating the intersection calculated in the intersection calculation step on information indicating a surface constituting the inside of the patient acquired in the patient shape acquisition step;
Surgery support information display method including On the first coordinate axis, a first surface showing a surface constituting the inside of a patient into which an endoscope that is inserted into the patient and images the inside is inserted, and a first three-dimensional shape of the surface of the patient are shown. Patient shape acquisition function to acquire the information of,
A surface image acquisition function for acquiring an image of the surface of the patient imaged when the endoscope is inserted into the patient;
A surface shape calculation function for calculating second information indicating a second three-dimensional shape of the surface of the patient on a second coordinate axis from the image of the surface of the patient acquired by the surface image acquisition function;
Of the first three-dimensional shape indicated by the first information acquired by the patient shape acquisition function and the second three-dimensional shape indicated by the second information calculated by the surface shape calculation function A plurality of reference points are provided on one three-dimensional shape, and for each of the plurality of reference points, the corresponding point at the shortest distance from the reference point is calculated from the limited range of the other three-dimensional shape, A coordinate axis that matches the first coordinate axis and the second coordinate axis by matching the first information and the second information based on the positional relationship between the reference point and the corresponding points. Match function,
From the image acquired by the surface image acquisition function, a half line indicating the optical axis that is the imaging direction of the endoscope on the second coordinate axis matched with the first coordinate axis by the coordinate axis matching function is calculated. Endoscope optical axis calculation function,
The inside of the patient related to the first line acquired by the half line indicating the optical axis that is the imaging direction of the endoscope calculated by the endoscope optical axis calculating function and the patient shape acquiring function is configured. An intersection calculation function for calculating an intersection with a surface;
An output function for superimposing and outputting information indicating the intersection calculated by the intersection calculation function on information indicating a surface constituting the inside of the patient acquired by the patient shape acquisition function;
A surgery support information display program for causing a computer to execute.