JP2011104079A - Medical image processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image supporting the positioning operation of an endoscope since the visual field range of a conventional endoscopic image is small, so that there is a desire to see peripheral tissues and since when generating a synthetic image by synthesizing the endoscopic image and an ultrasonic image, it is desired to position both of the images by a simple constitution. <P>SOLUTION: A medical image processing system is composed of an ultrasonograph 10 and an endoscopic apparatus 12. The endoscopic apparatus 12 forms the endoscopic image as a conventional one. The ultrasonograph 10 obtains volume data and identifies a plurality of high luminance points included in the data. The high luminance points express a plurality of spheres of reflection provided at the distal end of the endoscope 42. Since the three-dimensional position of the distal end can be identified from the high luminance points, the ultrasonic image as a projection image is formed from the volume data on the basis of the three-dimensional position. The endoscopic image which expressed a local area is superimposed on the ultrasonic image covering a wide area, and a synthetic image like this is displayed. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a medical image processing system, and more particularly to a medical image processing system having a function of combining an endoscopic image and an ultrasonic image.

  The endoscope device includes an endoscope. An observation means such as a CCD having a visual field in front is provided at the distal end of the endoscope. In addition, a light emitting device, an opening for projecting the surgical tool, and the like exist at the tip. In general, since the visual field range of the observation means is small, it is difficult or troublesome to find the target tissue while viewing the endoscopic image. For example, for a disease called twin-to-female transfusion syndrome, an endoscope is inserted into the uterus, and the image is observed with the endoscope. Surgery is performed to block the placental blood flow anastomosis. At that time, since the field of view of the endoscope sensor is, for example, about 10 to 20 mm, it is generally difficult to search and specify the anastomosis portion on the placenta only from the endoscopic image, and it takes time. is there.

  On the other hand, recently, three-dimensional ultrasonic diagnostic technology is being put into practical use. According to this technology, volume data is obtained by transmitting and receiving ultrasonic waves to and from a three-dimensional region in a living body, and an integrated image, a volume rendering image, and the like for a body tissue can be formed based on the volume data. . It is desired to make use of this technique for diagnosis and treatment using an endoscope.

  Patent Document 1 discloses a system for assisting by tomographic imaging in an operation using an endoscope (or laparoscope). This document describes an ultrasonic diagnostic apparatus as an example of an apparatus for obtaining a tomographic image. In the system, when a point of interest is specified by an operator on an endoscopic image, the relative position of the endoscope reference point with respect to the point of interest is obtained, while the absolute position of the endoscope reference point is obtained. The absolute position of the target point is obtained from the relative position and the absolute position. The apparatus described in Patent Document 1 only calculates a tomographic position for an observation target of an endoscope.

  Patent Document 2 describes an ultrasonic endoscope apparatus. In this apparatus, an image obtained by an endoscope and an image obtained by ultrasonic diagnosis are displayed in association with each other, and a cross-correlation calculation is used for the association. In this apparatus, the endoscope is provided with a CCD camera and an ultrasonic transducer, and three-dimensional ultrasonic data is acquired by manual scanning by pulling the endoscope forward. This apparatus merely spatially associates the three-dimensional surface shape data obtained by the endoscope with the three-dimensional surface shape data obtained by the ultrasonic diagnosis, and does not provide a support image for positioning the endoscope. .

Japanese Patent Laid-Open No. 10-113333 JP 2004-358096 A

  As described above, since it is difficult to specify a target region only from an endoscopic image with a narrow field of view, such an endoscopic image with a narrow field of view and an ultrasonic image with a wide field of view are synthesized, and the synthesized image is obtained. It is desired to provide the surgeon with a support image for positioning the endoscope. However, since the endoscope apparatus and the ultrasonic diagnostic apparatus are usually configured separately, their coordinate systems are independent of each other. If a large positioning means is added to each device in order to align and synthesize an endoscopic image and an ultrasonic image, the configuration of the entire system becomes complicated. It is desired to form the operation support image with a simple configuration.

  An object of the present invention is to provide an image that supports an endoscope positioning operation.

  Another object of the present invention is to enable generation of an image that supports an endoscope positioning operation with a simple configuration.

  The system according to the present invention has a distal end portion including an ultrasonic reflector, and transmits and receives ultrasonic waves to and from an endoscope inserted into the body and a three-dimensional space including the distal end portion of the endoscope. An ultrasonic probe, and coordinate calculation means for calculating three-dimensional coordinate information about the tip based on an ultrasonic reflector specific echo included in volume data obtained by transmitting and receiving ultrasonic waves to and from the three-dimensional space; Based on the three-dimensional coordinate information, an ultrasonic image forming unit that forms an ultrasonic image using the volume data, and the ultrasonic image and an endoscopic image acquired using the endoscope are combined. And an image composition means for generating a composite image.

  According to the above configuration, since the ultrasonic reflector is provided at the distal end portion of the endoscope, the distal end portion is inserted into a three-dimensional space (three-dimensional echo data capturing sky) formed by the ultrasonic probe. Then, the reflected wave from the ultrasonic reflector is observed as a specific echo. In other words, an ultrasonic reflector is provided in advance in a known part of the endoscope so that it can be distinguished from the body tissue and the endoscope body, or under the condition that it can be identified by an echo signal. It is done. If the ultrasonic reflector specific echo can be identified, the position of the tip in the ultrasonic transmission / reception coordinate system can be easily specified. That is, three-dimensional coordinate information can be obtained. It is preferably relative coordinate information, that is, the relative position of the tip with respect to the volume data. Therefore, it is not necessary to measure the position and posture of the ultrasonic probe, and it is not necessary to measure the position and posture of the endoscope. As a result, the system configuration can be greatly simplified. The composite image is a composite of an ultrasonic image and an endoscopic image (optical image). The synthesis is preferably performed in real time. Although it was difficult to identify the affected area without observing a wide area with the endoscopic image alone, the high-frequency ultrasound image is displayed as a background image, so it is easier to identify the affected area by observing it. . Since an image supporting the operation of the endoscope or the surgical instrument can be provided, the safety of the diagnostic treatment can be improved and the burden on the user can be greatly reduced.

  Preferably, the ultrasonic reflector is composed of a plurality of reflecting elements provided at the tip. By using a plurality of reflective elements (particularly, three or more reflective elements spaced apart from each other), it is possible to specify both the position and the posture of the tip portion. Preferably, the plurality of reflective elements are provided on an annular path around the central axis of the tip, and constitute a reflective element array. According to this configuration, even if some of the reflective elements are hidden behind the endoscope body, the remaining reflective elements can be specified. Preferably, the reflective element array is composed of six or more reflective spheres. According to this configuration, basically, three or more reflecting spheres can be basically observed regardless of the position and posture of the tip. It is desirable that each reflecting sphere (strongly reflecting element) protrudes from the surface of the endoscope body. However, if contact with tissue or a guide cylinder member becomes a problem, each reflecting sphere can be embedded. is there. If a sphere is used, a reflected wave can be reliably obtained no matter which direction the ultrasonic wave arrives. Note that an ultrasonic absorption layer may be provided on the outer surface of the endoscope so that each reflecting sphere can be observed more prominently.

  Preferably, the coordinate calculation means includes extraction means for extracting a plurality of reflection element specific echoes as the ultrasonic reflector specific echoes from the volume data, and a plurality of cubics for the plurality of reflection element specific echoes. Analyzing means for calculating the three-dimensional coordinate information based on the original position. The echo from the strong reflector is a high-intensity echo and can be easily distinguished from the tissue or the like in the volume data.

  Preferably, the extraction means includes a high-intensity echo extraction means for extracting a plurality of high-intensity echoes from the volume data, and the high-intensity echoes based on a mutual positional relationship between the plurality of high-intensity echoes. Reflection element specific echo extraction means for identifying whether or not is a reflection element specific echo. In order to exclude high luminance noise, it is desirable to refer to the distance between high luminance echoes. That is, it is possible to determine a high-intensity echo separated by a certain distance or more as noise. Here, the distance is defined in a three-dimensional space.

  Preferably, the high-intensity echo extraction unit extracts the plurality of high-intensity echoes by comparing each determination value with each voxel data constituting the volume data while gradually decreasing the determination threshold value. Such processing makes it possible to easily specify three or more high-intensity echoes depending on the situation.

  Preferably, the reflection element specific echo extraction means identifies the reflection element specific echo based on a distance between high-intensity echoes. Preferably, the reflection element specific echo extraction unit extracts three or more reflection element specific echoes that satisfy a condition in which the distance between the high-intensity echoes is included between an upper limit value and a lower limit value.

  The system which concerns on this invention has the front-end | tip part provided with the ultrasonic reflector as a strong reflector, and an endoscopic means, the endoscope inserted in a body, and the signal acquired using the said endoscope An endoscopic image forming means for forming an endoscopic image based on the ultrasonic probe, an ultrasonic probe that is provided outside the body and transmits / receives ultrasonic waves to / from a three-dimensional space including the distal end of the endoscope, and the three-dimensional Coordinate calculation means for calculating three-dimensional coordinate information about the tip based on an ultrasonic reflector specific echo contained in volume data obtained by transmitting and receiving ultrasonic waves to and from space, and based on the three-dimensional coordinate information Means for forming an ultrasound image using the volume data, the ultrasound image forming means for forming a projection image covering the field of view of the endoscopic image as the ultrasound image, and the ultrasound image And the endoscopy Is characterized in that comprises an image synthesizing means for generating a composite image by synthesizing the door, and a display means for providing the composite image to a person operating the endoscope.

  Preferably, the ultrasonic image forming unit is configured to set a line-of-sight group that covers the visual field of the endoscopic unit with respect to the volume data, and from the distal end side along each line of sight constituting the line-of-sight group. Means for calculating a pixel value for each line of sight by executing rendering in front thereof, thereby forming the ultrasonic image. Examples of rendering methods include volume rendering and cumulative projection. An image in which the echo level corresponds to the luminance may be formed, or an image based on Doppler information may be formed.

  Desirably, the said image composition means produces | generates the said synthesized image by superimposing the endoscopic image as a narrow area image on the said ultrasonic image as a wide area image. Preferably, the endoscope is a therapeutic instrument that is inserted into the uterus for the treatment of a twin-to-twin transfusion syndrome and has a laser emission function.

  ADVANTAGE OF THE INVENTION According to this invention, the image which assists positioning operation of an endoscope can be provided. Alternatively, an image that supports the positioning operation of the endoscope can be generated with a simple configuration.

1 is a block diagram showing a preferred embodiment of a medical image processing system according to the present invention. It is a conceptual diagram which shows a three-dimensional space (real space). It is a figure for demonstrating the setting of an observation origin and a reference direction. It is a figure for demonstrating the setting of a process start surface. It is a figure for demonstrating the setting of several eyes | visual_axis. It is a figure which shows an example of an ultrasonic image. It is a figure for demonstrating the binarization process with respect to an endoscopic image. It is a figure which shows the ultrasonic image by which the binarization process was carried out. It is a figure which shows an example of a synthesized image (endoscope operation assistance image). 3 is a flowchart illustrating processing contents of an ultrasonic image forming unit illustrated in FIG. 1. It is a flowchart which shows the content of the image synthetic | combination part shown in FIG. It is a front view which shows the front-end | tip part (especially strong reflector row | line | column) of an endoscope. It is a perspective view which shows the front-end | tip part (especially strong reflector row | line | column) of an endoscope. It is a conceptual diagram which shows the result of having performed the high-intensity point extraction process, changing a threshold value in steps. It is a flowchart which shows the process which extracts three high-intensity points for calculating | requiring an observation origin. It is a figure for demonstrating the discrimination condition of a high-intensity point and noise. It is a figure for demonstrating the calculation method of a center point (observation origin).

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the drawings.

(1) Description of Overall System FIG. 1 shows a preferred embodiment of a medical image processing system according to the present invention, and FIG. 1 is a block diagram showing the overall configuration thereof. The system shown in FIG. 1 is a system that combines endoscopic treatment and ultrasound diagnosis, and the target tissue in this embodiment is the placenta in the uterus. Of course, the system according to the present invention can be used for other tissues.

  In FIG. 1, the system according to the present embodiment is roughly composed of an ultrasonic diagnostic apparatus 10 and an endoscopic apparatus 12. However, it is also possible to entrust the image processing function to an external computer or the like. In this case, the present system is configured by an ultrasonic diagnostic apparatus, an endoscopic apparatus, and a computer. Various configurations are conceivable for realizing the system according to the present embodiment.

  The ultrasonic diagnostic apparatus 10 will be described. The probe 14 is a transmitter / receiver that transmits / receives an ultrasonic wave. In this embodiment, a 3D probe that transmits / receives an ultrasonic wave to / from a three-dimensional space in a living body is used. Specifically, the probe 14 includes a probe head, a probe cable, a probe connector, and the like. An array transducer 16 is provided in the probe head. The array transducer 16 is a 2D array transducer in the present embodiment, and an ultrasonic beam is formed by the 2D array transducer 16, and the ultrasonic beam is two-dimensionally scanned to form a three-dimensional data capture space. Is done. Volume data is acquired from within the three-dimensional space. Of course, a three-dimensional space may be formed by mechanically scanning the 1D array transducer.

  The transmission / reception unit 26 includes a transmission beam former as a transmission unit and a reception beam former as a reception unit. At the time of transmission, a plurality of transmission signals are supplied in parallel from the transmission / reception unit 26 to the array transducer 16. As a result, a transmission beam is formed in the array transducer 16. At the time of reception, the reflected wave from the living body is received by the array transducer 16, and a plurality of reception signals are output from the array transducer 16 to the transmission / reception unit 26 in parallel. In the transmission / reception unit 26, a phasing addition process is applied to a plurality of reception signals, thereby obtaining a reception signal after phasing addition, that is, beam data.

  The beam data is output to the signal processing unit 28. The signal processing unit 28 includes various circuits such as a detection circuit and a logarithmic compression circuit, and executes predetermined signal processing on the beam data. The processed beam data is stored in the 3D memory 30. Coordinate transformation is applied at the time of writing, and echo data (voxel data) constituting each beam data is stored at the corresponding address. Of course, coordinate transformation may be applied at the time of reading. The coordinate conversion generally means conversion from a transmission / reception wave coordinate system to a memory space coordinate system. As a result, volume data is stored in the 3D memory 30.

  In the present embodiment, a three-dimensional echo data capturing space, that is, a three-dimensional space, is formed so as to include the local tissue observed by the endoscopic device 12, and conversely, at such an appropriate position. The 3D probe 14 is positioned so that a three-dimensional space is formed. Accordingly, the 3D memory 30 stores volume data corresponding to a wide area organization including a local tissue that is an object of endoscopic imaging.

  The coordinate calculation unit 102 calculates coordinate data necessary for rendering by applying the processing shown in FIGS. 14 and 15 later to the volume data. The coordinate data is information for specifying the endoscopic observation origin and the visual field center direction. Rendering conditions for forming an ultrasonic image are determined based on such coordinate data. The specific calculation contents in the coordinate calculation unit 102 will be described in detail later with reference to the drawings after FIG.

  The ultrasonic image forming unit 32 forms an ultrasonic image as a projection image by performing rendering processing based on partial data (partial volume data) in the volume data. In that case, data processing is executed so that an ultrasonic image is formed from the same direction as the observation point, that is, the viewpoint in the endoscopic image. In that case, the coordinate data output from the relative coordinate calculation unit 102 is referred to.

  The image synthesizing unit 34 is a module that synthesizes an endoscopic image described later on an ultrasonic image. As a synthesis method, a superposition method and a fitting method are conceivable. In the superposition method, the endoscopic image is partially overlapped on the ultrasonic image. When the fitting method is employed, the region where the endoscopic image is fitted is removed from the ultrasonic image, that is, a hole is formed, and the endoscopic image is fitted into the vacant portion. Of course, various conventionally known methods can be employed as the image composition method. In the present embodiment, in the synthesis of two images, the correlation calculation is sequentially performed while changing the relative positional relationship between the two in a stepwise manner. The relationship is certified. If the two images are combined under such an appropriate positional relationship, the positional deviation between the images can be minimized, and the distortion between the images can be reduced. However, such correlation calculation only needs to be executed as necessary, and a certain effect can be obtained by simply superimposing two images.

  As described above, when a composite image is generated in the image composition unit 34, the image data is sent to the display unit 36, and the composite image is displayed on the screen of the display unit 36. The composite image is a new image in which an optical image and an acoustic image are combined. Specifically, an ultrasonic image is displayed around an optical image similar to the conventional one, and was not visible in the past. Since it is possible to recognize the structure and state of the peripheral part through an ultrasonic image, it is possible to provide support for endoscopic operation and to quickly detect the affected part. The endoscopic image can be processed and modified prior to image synthesis, and this is the same for the ultrasonic image. For example, edge enhancement processing, contrast enhancement processing, or the like may be performed.

  The main control unit 38 performs operation control of each component included in the ultrasonic diagnostic apparatus 10, and the main control unit 38 is specifically configured by a CPU and an operation program. An operation panel 40 is connected to the main control unit 38, and the user can set operating conditions and input parameters using the operation panel 40. Incidentally, the ultrasonic image forming unit 32 and the image synthesizing unit 34 can be realized as software functions, and such program processing can be performed in the ultrasonic diagnostic apparatus 10, but volume data is stored. It is also possible to transfer to an external PC and perform image processing on the external PC. However, it is desirable that the composite image is updated in real time in support of the endoscope operation, and it is desirable to construct a system that can realize such real time performance. In the present embodiment, the endoscopic device 12 is combined with the ultrasonic diagnostic apparatus 10, that is, the image data from the endoscopic apparatus 12 is used in the ultrasonic diagnostic apparatus 10 and displayed in real time. An endoscopic image that is also displayed in real time on the ultrasonic image is synthesized.

  Next, the endoscopic device 12 will be described. Reference numeral 42 denotes an endoscope. The endoscope 42 includes a part inserted into the body and a part positioned outside the body. An image sensor 44, a light emitter 46, and the like are provided in the body part. The image sensor 44 is a CCD camera, for example. Of course, other image acquisition devices may be provided. The light emitter 46 is a device for illuminating the front when capturing an image. However, the light emitter 46 can be omitted by using, for example, a high-sensitivity infrared sensor. One or a plurality of channels (not shown) are provided in the endoscope 42, and a surgical instrument is inserted using any of the channels to treat a blood vessel or the like running on the surface of the placenta. Can be done.

  An endoscopic image forming unit 47 is provided in the endoscopic device main body 45, and the endoscopic image forming unit 47 forms an endoscopic image as an optical image based on a signal output from the image sensor 44. The image data is output to the image composition unit 34 as described above.

  A three-dimensional space 60 is shown in FIG. This three-dimensional space 60 is a region (real space) where ultrasonic waves are transmitted and received, and corresponds to the volume data described above. The three-dimensional space 60 is configured by electronic scanning of an ultrasonic beam in the probe head 56. Although a cubic three-dimensional space 60 is shown in FIG. 2, the shape is not limited to a cube. For example, in the method of mechanically swinging and scanning a convex 1D array transducer, a pyramid three-dimensional space is formed.

  The three-dimensional space 60 includes a part of the uterus in this embodiment, and a part of the placenta 62 is included therein as shown. Reference numeral 62A represents the surface of the placenta, and a plurality of blood vessels 64 run on the surface (surface layer) 62A.

  The endoscope 58 is configured as a hard rod-like member in the present embodiment, and the distal end portion of the endoscope 58 is inserted into the uterus through a hole formed in the abdomen. A joint portion having flexibility may be provided at the distal end portion of the endoscope 58 or other portions. The above-described image sensor is provided at the distal end portion of the endoscope 58, and a visual field range by the image sensor is indicated by reference numeral 66. As shown in the figure, the visual field range 66 is a very small area on the placenta surface 62A, and it is not easy to detect the diseased part through the visual field range. That is, there is a need to obtain information about the state of the tissue existing around the visual field range 66. Therefore, in the present embodiment, as described above, a combination of an endoscopic image and an ultrasonic image is realized.

  An example of creating an ultrasound image will be described with reference to FIGS. In FIG. 3, in this embodiment, first, an observation origin 58A corresponding to a viewpoint on the image sensor, that is, a center point, in the endoscope 58 is specified. The identification is performed by the coordinate calculation unit 102 shown in FIG. The three-dimensional position of the observation origin 58A is specified according to the three-dimensional coordinate system of the data space where the volume data exists. In addition, even if the center of the actual observation window and the center of the tip portion are deviated, in other words, even when only the center of the tip portion can be specified instead of the center of the actual observation window, the above-described matching processing is performed. Is executed, there is no problem in the synthesis of the two images. Along with the observation origin 58A, a reference direction extending from the front of the visual field, that is, the visual field direction 70 is specified. The identification is also performed in the relative coordinate calculation unit 102 shown in FIG. 1, but the calculation may be performed in the ultrasonic image forming unit 32. The visual field direction 70 corresponds to the central axis of the visual field range 66 or corresponds to the axial direction of the endoscope 58.

  Next, as shown in FIG. 4, a processing start surface 72 is set with reference to the observation origin 58A. Specifically, the processing start surface 72 is set on the three-dimensional storage space of the 3D memory. In the present embodiment, the processing start surface 72 is a surface that passes through the observation origin 58 </ b> A and is orthogonal to the reference direction 72. Although the size of the processing start surface 72 can be arbitrarily set, it is desirable that the size is determined so as to cover at least the periphery of the visual field range 66, and it is particularly desirable that the processing start surface 72 passes through the entire volume data. It is determined as a range having a size of. The processing start surface 72 may be set not at the surface passing through the observation origin 58A but at a position ahead of it. That is, in this embodiment, there is amniotic fluid between the placenta surface and the image sensor, and this amniotic fluid is a part that is hardly reflected in the ultrasonic image, so the processing start surface is set forward as necessary. It is possible to do. Of course, it is also possible to set the processing start surface behind.

  Next, as shown in FIG. 5, a line-of-sight (ray) group 74 is set as a direction orthogonal to the processing start surface 72 as a reference. The line-of-sight group 74 includes a plurality of lines of sight 76. In the present embodiment, the plurality of lines of sight 76 are parallel to each other, but the plurality of lines of sight 76 may be set in a non-parallel relationship. For example, a plurality of lines of sight 76 may be set radially from the observation origin 58A. Then, rendering is performed for each line of sight, that is, a pixel value is determined for each line of sight. By mapping these pixel values on a two-dimensional plane, an ultrasonic image can be constructed. An example of the rendering method is a cumulative projection method, and it is particularly desirable to apply the volume rendering method. In the present embodiment, rendering conditions are determined so that not only the surface layer portion of the placenta but also the inside of the placenta is reflected in the ultrasound image.

  An ultrasonic image formed as a result of such rendering processing is schematically shown in FIG. In FIG. 6, a wavy line circle 84 represents the field of view range by the endoscope. It can be understood that the periphery of the range is widely imaged. There is a blood vessel group 78, and the blood vessel group 78 includes a blood vessel 80 running on the surface layer and a blood vessel 82 running inside. Of course, it is possible to freely change the rendering depth by changing the rendering conditions.

  As described above, when an ultrasound image is configured, the ultrasound image and the endoscopic image are combined, but in this embodiment, correlation calculation is performed between the two images as described above. Has been. That is, the matching process is performed so that the two images are spatially aligned. For this purpose, each image is binarized.

  In FIG. 7A, an endoscopic image 86 is shown. The image is a circular image corresponding to the visual field range. A blood vessel 88 appears inside thereof. A binarization process is performed on this, and a binarized image 90 generated thereby is shown in FIG. 7B. In the binarization process, 1 (or 0) is given to the blood vessel part, and 0 (or 1) is given to the other tissue parts.

  On the other hand, binarization processing is also performed on the ultrasonic image. Specifically, a binarization process is performed on the ultrasound image 76 shown in FIG. 6 using a threshold value for discriminating blood vessels and tissues, and the binary image shown in FIG. This is a converted image 96. A wavy line circle 84 represents the visual field range of the endoscopic image. In the present embodiment, correlation calculation is performed while the positional relationship between the binarized image 96 shown in FIG. 8 and the binarized image 90 shown in FIG. It is repeatedly executed, and it is recognized as an appropriate positional relationship when the best correlation value is obtained. Then, the two ultrasonic images (image before binarization) and the endoscopic image (image before binarization) that are in the proper positional relationship are synthesized, thereby generating a synthesized image. An example is shown in FIG. The composite image 100 is an image obtained by superimposing the endoscopic image 86 on the ultrasonic image 76. In this embodiment, the ultrasonic image 76 is a black and white image, and the endoscopic image 86 is a color image. Since the endoscopic image 86, which is an optical image, is generally clearer, in the synthesized image 100, a state in which the tissue clearly emerges with respect to the background in the circular area of interest is manifested. become. If the endoscope is moved, the visual field range is moved, so that the content of the endoscopic image 86 is updated in real time. In this case, the position of the endoscopic image 86 may be changed on the ultrasonic image 76, or the ultrasonic image 76 is moved or the content thereof is moved in a state where the screen display position of the endoscopic image 86 is fixed. You may make it update. The size of the endoscopic image may be dynamically changed according to the distance by measuring the distance between the image sensor and the tissue. In the present embodiment, even if discontinuity or distortion occurs between two images, an image representing a tissue structure can be displayed around the endoscopic image 86 with a certain degree of certainty. There is an advantage that it is possible to support, and the identification of the affected part can be facilitated. That is, what is actually focused on is the endoscopic image 86, and if the visual field range is moved, a clear part can be moved, and the image existing in the vicinity is only for assistance. Therefore, even if there is some misalignment between the two images, it is possible to improve convenience compared to the conventional case.

  Next, an example of the operation of the system will be described with reference to FIGS. FIG. 10 shows an operation example of the ultrasonic image forming unit 32 shown in FIG. 11, and FIG. 11 shows an operation example of the image composition unit 34 shown in FIG.

  First, referring to FIG. 10, in S101, the observation origin and the reference direction are calculated based on a method that will be described in detail later with reference to FIGS. Specifically, the center point of the distal end portion of the endoscope and the forward direction therefrom (the forward central axis direction as viewed from the probe) are specified. In S102, the processing start surface is specified as shown in FIG. If the processing start surface is obtained first, the specification of the reference direction can be omitted. In S103, as shown in FIG. 5, a plurality of lines of sight extending from the processing start surface are set, and by performing a rendering operation along each line of sight (ray), processing for the partial volume data is executed. An ultrasonic image is formed as a rendering image reflecting the data. An example of the ultrasonic image is as shown in FIG.

  Next, referring to FIG. 11, in S201, the endoscopic image obtained by the endoscopic device is binarized, thereby generating a binarized endoscopic image. Similarly, in S202, the ultrasonic image is binarized, thereby generating a binarized ultrasonic image. In S203, both are synthesize | combined temporarily, making the center point of an endoscopic binarized image correspond with the reference axis point of an ultrasonic binarized image. Here, the reference axis point is a point corresponding to the reference axis in the ultrasonic image, and by aligning the central point of the endoscopic binarized image with this point, the two images are roughly aligned in the three-dimensional space. It is possible to make it. Based on the initial state, the search width in the correlation calculation is set as follows.

  In S204 to S206, the correlation calculation is executed while changing the scale ratio of the image. Specifically, one image (preferably an endoscopic binarized image) is enlarged or reduced, and a correlation value is calculated between the two images in S205. Then, until the correlation value is regarded as the maximum in S206, the correlation calculation while changing the scale ratio of the one image is repeatedly executed. When it is determined in S206 that the correlation value is the maximum, correlation value calculation processing is performed while performing parallel movement in S207 to S209.

  Specifically, in S207, one image is translated in the vertical and horizontal directions, and in S208, a correlation value is calculated in the state after the translation. Then, while repeating the parallel movement, the correlation value calculation is repeatedly executed until the correlation value is regarded as the maximum in S209. Even if it is determined in S209 that the correlation value is the maximum, the processes of S210 to S212 are executed. Specifically, one image is rotated in S210, and a correlation value is calculated in S211. This process is repeatedly executed until it is determined in S212 that the correlation value is maximum. If it is determined in S212 that the correlation value is maximum, S213 is executed.

  In S213, even if it is determined whether the correlation value is saturated or not from the transition so far, if it is not saturated, each step after S204 is repeatedly executed. That is, the best matching state of the two images is searched until the correlation value is saturated while changing the combination of conditions. When it is determined that the correlation value is saturated in S213, the ultrasonic image and the endoscopic image are combined and displayed on the screen in S214. In FIG. 11, if three changes of enlargement / reduction processing, parallel movement processing, and rotation processing are included, one or two of them may be implemented. However, by including the enlargement / reduction processing in particular, even if the distance between the image sensor and the object surface is unknown, there is an advantage that the scale of the two images can be matched to construct a composite image without a sense of incongruity. is there.

  In the above configuration, since an ultrasonic image in which a three-dimensional region is imaged is used, it is possible to display a blood vessel that travels inside the tissue, thereby using a surgical instrument to treat the blood vessel. There is an advantage that the safety can be improved even in the case of performing. In this embodiment, the synthesized image is provided to the operator in real time, and the treatment site on the placenta can be quickly identified while viewing the composite image. In other words, the endoscope can be appropriately positioned by operating the endoscope. Since the periphery of the affected area can be observed at a time, it is possible to prevent the tip from inadvertently contacting the tissue. If a treatment site is specified, a laser beam is irradiated to the treatment site.

(2) Description of Relative Coordinate Calculation Method Using Reflector Next, a relative coordinate calculation method using a reflector will be described in detail with reference to FIGS.

  12 and 13 show the distal end portion of the endoscope 42. FIG. 12 is a plan view, and FIG. 13 is a perspective view. An observation window 106, an irradiation window 108, and a forceps port 110 for optical observation are provided on the distal end surface 42A of the distal end portion. Although the center of the distal end surface 42A and the center of the observation window 106 do not coincide with each other, since the matching process is applied at the time of image synthesis, the positional deviation does not become a problem. The observation window 106 is composed of a window member for capturing an optical image by an image sensor. The irradiation window 108 is constituted by a window member for irradiating light generated by the light emitter forward. The forceps port 110 constitutes an end portion of a channel through which the instrument is inserted, and a laser treatment device for blood vessel ablation is inserted therein. The tip surface 42A is a circular surface, and a reflector 112 is provided on the periphery (periphery). Specifically, the reflector 112 constitutes a ring-shaped reflective sphere array composed of a plurality of reflective spheres (reflective elements) 114. Each reflecting sphere 114 is composed of a microsphere (porous) made of a special material that strongly reflects ultrasonic waves. In this embodiment, eight reflecting spheres 114 are provided, and their mutual interval (mutual angle) is a constant pitch (equal angle). In general, if six or more reflecting spheres are provided, echoes from at least three reflecting spheres can be observed regardless of the position of the probe. That is, even if some of the reflecting spheres are hidden behind the endoscope body, it is possible to always observe echoes from other reflecting spheres. Since the mutual positional relationship is known, if the three-dimensional positions of three or more reflecting spheres can be specified, the position of the tip surface 42A, specifically the center position thereof, in the three-dimensional space can be calculated. Become. The axial direction (reference direction) extending forward from the center position can be defined as a direction that leaves the center of the tip surface and moves away from the probe. Echoes from the endoscope body are also observed, but an ultrasonic absorption layer is provided on the outer surface of the endoscope body in order to reduce such echoes and make the echoes from each reflecting sphere appear. You may do it.

  FIG. 14 shows a high-luminance point extraction process as a conceptual diagram. (A) shows a three-dimensional space (real space) 60. This corresponds to an ultrasonic wave transmission / reception space by the probe head 56. The distal end portion of the endoscope 58 is located in the three-dimensional space 60. An ultrasonic reflector (reflecting sphere array) 112 is provided at the distal end portion of the endoscope 58. (B) to (F) show a three-dimensional space (data space) 116 corresponding to volume data. The threshold value α is a threshold value used when binarizing each echo data (voxel data). In the example shown in (B), α1 is set as the threshold value α, and (B) shows the result of binarization processing using such a threshold value α. Here, one high luminance point is extracted as indicated by reference numeral 118. This high luminance point corresponds to one of the reflecting spheres. (C) shows a processing result when the threshold value α is lowered by one step to α2 and binarization processing is performed using the threshold value α. As indicated by reference numeral 120, two high luminance points corresponding to two reflecting spheres are extracted. Since the tissue is relatively low in brightness, it is not extracted at this stage. (D) shows a processing result when the threshold value α is further lowered by one step to α3 and binarization processing is performed using the threshold value α. As indicated by reference numeral 120, as described above, two high-intensity points are extracted. On the other hand, as indicated by reference numeral 122, a high-intensity point corresponding to high-intensity noise is extracted. (E) shows a processing result when the threshold value α is further lowered by one level to α4 and binarization processing is performed using the threshold value α. As indicated by reference numeral 124, three high luminance points corresponding to the three reflecting spheres and one high luminance point corresponding to noise are extracted. As described above, when the threshold value α is lowered step by step, the number of high luminance points corresponding to the reflecting spheres increases, but on the other hand, there is a high possibility that high luminance noise will be hit. .

  Therefore, in this embodiment, as will be described in detail later, the distance between the high luminance points (the distance in the three-dimensional space) is measured, and the distance is high when the distance is greater than or equal to a predetermined value. It is determined that it is luminance noise. For example, when there are three high-intensity points, the distance between two specific high-intensity points satisfies a certain condition, and one specific high-intensity point is compared with the other two high-intensity points. If the distance between them does not satisfy a certain condition, it is determined that the high luminance point is noise. As a result of applying such noise removal processing, a result as shown in (F) is obtained. That is, the three-dimensional space 116 includes only three high-luminance points (specific echoes) corresponding to three reflecting spheres, and excludes high-luminance points corresponding to noise. Of course, four or more high-luminance points corresponding to four or more reflecting spheres may be extracted. If at least three high-luminance points can be extracted, the tip surface of the tip portion can be spatially defined by these three points, and at the same time, the center point can be defined. The reference direction is defined as the forward direction passing through the center point.

  FIG. 15 is a flowchart showing a process for specifying three (or more) reflecting spheres.

  In S301, a maximum value is set as the threshold value α as an initial setting. In S302, binarization processing is applied to the volume data using the threshold value α. That is, voxel data exceeding the threshold value α is converted to 1, and the other voxel data is converted to 0. In S303, for example, thinning processing is applied as post-processing after binarization. This thinning process is performed in order to specify the center point as one point when the high luminance point has a spread. However, this processing may be executed when necessary. In S304, when there are two or more voxels having data of value 1, the distance between such voxels is calculated. In S305, it is determined whether the calculated distance satisfies the predetermined condition or not, and if it is determined that the high-intensity point corresponding to the noise is based on the determination result, it is excluded (erased). The The discrimination method based on the distance will be described later with reference to FIG. The process of S305 is normally performed when three or more high-luminance points are obtained. Of course, methods other than those based on distance may be applied as the noise removal processing method. For example, it is conceivable to use the size and shape of a high-luminance point as an identification criterion. In S306, the number of high luminance points is counted, and if the count value is 3 or more, the process proceeds to S3008, otherwise the process proceeds to S302. That is, the above process is repeatedly executed while decreasing the threshold value α step by step. In S308, the coordinates of the three high-intensity points having a close distance satisfying the predetermined condition are calculated and output. That is, the process moves to S101 shown in FIG. The tip surface is defined from the coordinates of the three points, and at the same time, the three-dimensional position of the center is specified, and the reference direction is specified as a perpendicular extending from there.

  FIG. 16 shows the mutual positional relationship between the plurality of reflecting spheres 114 constituting the ultrasonic reflector 112. In the three-dimensional space, the interval between two adjacent reflecting spheres 114 is a, which is known. The longest distance between two reflecting spheres is b, which is also known. That is, the distance between the points is always a to b in the case of a regular high luminance point. Conversely, if the distance between two high brightness points does not satisfy the condition, one or both of them may be noise. If one of them has already been found to be a regular high luminance point, it can be determined that the other is noise. Incidentally, the intervals between the plurality of reflecting spheres may be uneven, and the rotational orientation of the tip portion around the central axis may be specified. In that case, the determination condition may be determined based on the actual arrangement of the reflecting spheres.

  FIG. 17 is a conceptual diagram showing a method for specifying the center point of the tip surface. Now, when the three-dimensional positions (three-dimensional coordinates) of the high-luminance points P1, P2, and P3 adjacent to each other are specified, there are two adjacent high-luminance point pairs, and two straight lines L1 passing through these pairs. , L2 is identified. The distance on each straight line L1, L2 is a. A midpoint of each straight line L1, L2 is specified, and perpendicular lines (orthogonal lines) L3, L4 passing through the midpoint are specified. A center point O is specified as the intersection between them. In the example shown in FIG. 17, three high-intensity points are arranged, but even if this is not the case, the center point can be easily set as a perpendicular line taking the midpoint with reference to two straight lines defined by the three points. Can be specified. The center point is a reference point for rendering. However, the position of the observation window may be specified from the center point, and the position may be used as the rendering reference point. If three points are determined, a front end surface can be defined, and it may be used as a rendering start surface or a rendering reference surface. In the present embodiment, since the plurality of reflecting spheres are arranged in an annular shape, basically at least three high-luminance points can be extracted regardless of the position of the probe.

Next, for reference, a method for obtaining a plane equation including three high-luminance points (three points in a three-dimensional space) will be described. The coordinates of the three points are as follows.

The plane to be obtained is defined by the following equation.

において、ｄ＝１とすると、上記（１）式は、以下のように表現される。

Specifically, three coefficients a, b, and c included in the above formula are obtained. Since d can be calculated by being included in the above three coefficients, it can be calculated as d = 1 for convenience. That means

Where d = 1, the above equation (1) is expressed as follows.

Since the above three points satisfy the above equation (3), a plane equation can be obtained by obtaining a, b, and c that satisfy the following simultaneous equations.

If this simultaneous equation is solved using the expression of the determinant, the expression (4) is expressed as follows.

A matrix K is obtained by multiplying both sides of the above equation (5) by the inverse matrix of the matrix P.

であるから、Ｋは、

により求めることが可能である。但し、以下の通りである。

here,

So K is

It is possible to obtain by However, it is as follows.

Therefore, the plane equation is obtained as follows.

  According to the configuration of the above embodiment, since the three-dimensional coordinates of the distal end portion of the endoscope can be specified in the transmission / reception coordinate system, a positioning device is provided in each of the probe and the endoscope to measure each absolute coordinate. This is not necessary, and the system configuration can be simplified and the positioning accuracy can be improved. Ultrasonic image generation and image synthesis are performed in real time for each volume scan, so the synthesized image can be provided to the surgeon and assist the operation of the endoscope, so that diagnostic treatment can be performed smoothly. It is possible to improve safety. Thus, this system produces a medically beneficial effect.

  DESCRIPTION OF SYMBOLS 10 Ultrasonic diagnostic apparatus, 12 Endoscopic apparatus, 14 Probe, 32 Ultrasonic image forming part, 34 Image composition part, 42 Endoscope, 44 Image sensor, 45 Endoscopic apparatus main body, 47 Endoscopic image forming part, 102 Coordinates Arithmetic unit.

Claims (13)

An endoscope having a tip with an ultrasonic reflector and inserted into the body;
An ultrasonic probe for transmitting and receiving ultrasonic waves to and from a three-dimensional space including the tip of the endoscope;
Coordinate calculation means for calculating three-dimensional coordinate information about the tip based on an ultrasonic reflector specific echo included in volume data obtained by transmitting and receiving ultrasonic waves to and from the three-dimensional space;
An ultrasonic image forming means for forming an ultrasonic image using the volume data based on the three-dimensional coordinate information;
Image synthesizing means for generating a synthesized image by synthesizing the ultrasonic image and an endoscopic image acquired using the endoscope;
A medical image processing system comprising: The system of claim 1, wherein
The medical image processing system, wherein the ultrasonic reflector is composed of a plurality of reflecting elements provided at the tip. The system of claim 2, wherein
The medical image processing system, wherein the plurality of reflective elements are provided on an annular path around a central axis of the tip, and constitute a reflective element array. The system of claim 3, wherein
The medical image processing system, wherein the reflective element array is composed of six or more reflective spheres. The system according to any one of claims 2 to 4,
The coordinate calculation means includes
Extracting means for extracting a plurality of reflection element specific echoes as the ultrasonic reflector specific echoes from the volume data;
Analysis means for calculating the three-dimensional coordinate information based on a plurality of three-dimensional positions of the plurality of reflection element specific echoes;
A medical image processing system comprising: The system of claim 5, wherein
The extraction means includes
High intensity echo extraction means for extracting a plurality of high intensity echoes from the volume data;
Reflection element specific echo extraction means for identifying whether each of the high intensity echoes is a reflection element specific echo based on the mutual positional relationship of the plurality of high intensity echoes;
A medical image processing system comprising: The system of claim 6, wherein
The high-intensity echo extraction unit extracts the plurality of high-intensity echoes by comparing each determination value with each voxel data constituting the volume data while gradually decreasing the determination threshold. Medical image processing system. The system according to claim 6 or 7,
The medical image processing system, wherein the reflection element specific echo extraction means identifies the reflection element specific echo based on a distance between high-intensity echoes. The system of claim 8, wherein
The reflection element specific echo extraction unit extracts three or more reflection element specific echoes that satisfy a condition in which the distance between the high-intensity echoes is included between an upper limit value and a lower limit value. system. An endoscope having an ultrasonic reflector as a strong reflector and a distal end portion provided with an endoscopic means, and inserted into the body;
An endoscopic image forming means for forming an endoscopic image based on a signal acquired using the endoscope;
An ultrasonic probe that is provided outside the body and transmits and receives ultrasonic waves to and from a three-dimensional space including the tip of the endoscope;
Coordinate calculation means for calculating three-dimensional coordinate information about the tip based on the ultrasonic reflector specific echo contained in the volume data obtained by transmitting and receiving ultrasonic waves to and from the three-dimensional space;
Based on the three-dimensional coordinate information, an ultrasonic image forming unit that forms an ultrasonic image using the volume data, and forms a projection image that covers a visual field area of the endoscopic image as the ultrasonic image. Means,
Image combining means for generating a combined image by combining the ultrasonic image and the endoscopic image;
Display means for providing the composite image to a person operating the endoscope;
A medical image processing system comprising: The apparatus of claim 10.
The ultrasonic image forming means includes:
Means for setting a line-of-sight group covering the field of view of the endoscopic means for the volume data;
Means for calculating a pixel value for each line of sight by performing rendering from the tip side to the front along each line of sight constituting the line of sight group, thereby forming the ultrasonic image;
A medical image processing system comprising: The apparatus of claim 10.
The medical image processing system, wherein the image synthesizing unit generates the synthesized image by superimposing an endoscopic image as a narrow area image on the ultrasonic image as a wide area image. The system according to any one of claims 10 to 12,
A medical image processing system, wherein the endoscope is a therapeutic instrument that is inserted into a uterus for treatment of a twin-to-twin transfusion syndrome and has a laser emission function.

Images