JP2010088699A - Medical image processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To meet the demand that the conventional endoscope image has a narrow range of view and should be enlarged so as to include more tissues around it. <P>SOLUTION: The medical image processing system includes an ultrasonic diagnostic apparatus 10 and an endoscope 12. The endoscope 12 will produce endoscopic images in a traditional way. In the ultrasonic diagnostic apparatus, arithmetic operation will be done on a processing start plane across the original point of observation and on a plurality of visual lines starting from the processing start plane. Then rendering will be done for the respective visual lines. By this, an ultrasonic image will be constructed. A coordinate operation will be done in reference to the coordinate data output from coordinate operation parts 24 and 54. For synthesizing the endoscopic image with the ultrasonic image, a correlative operation is used to determine the appropriate positional relationship of a user. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a medical image processing system, and more particularly to a system for synthesizing an endoscopic image and an ultrasonic image.

  The endoscope device includes an endoscope as an endoscope means. A sensor such as a CCD having a visual field in front (that is, the observation direction) is provided at the distal end portion of the endoscope. In addition, a light emitting device, an opening for projecting the surgical tool, and the like exist at the tip. In general, since the field of view of a sensor is small, there is a problem that it is difficult or troublesome to find a target tissue while viewing an endoscopic image. For example, for a disease called twin-to-fetal transfusion syndrome, an endoscope is inserted into the uterus, and an image is observed with the endoscope. Surgery is performed to block the placental blood flow anastomosis. At that time, since the field of view of the endoscope sensor is, for example, about 10 to 20 mm, it is generally difficult to search and specify the anastomosis portion on the placenta only from the endoscopic image, and it takes time. is there.

  On the other hand, recently, three-dimensional ultrasonic diagnostic technology is being put into practical use. According to this technique, volume data is acquired by transmitting and receiving ultrasonic waves to and from a three-dimensional region in a living body, and an integrated image, a volume rendering image, and the like for a body tissue can be formed based on the volume data. It is desired to utilize this technique for diagnosis and treatment using an endoscope.

Japanese Patent Laid-Open No. 10-113333 JP 2004-358096 A

  Patent Document 1 discloses a system for assisting by tomographic imaging in an operation using an endoscope (or laparoscope). This document describes an ultrasonic diagnostic apparatus as an example of an apparatus for obtaining a tomographic image. In the system, when a point of interest is specified by an operator on an endoscopic image, the relative position between the point of interest and the endoscope reference point is obtained, while the absolute position of the endoscope reference point is obtained. The absolute position of the point of interest is obtained from the relative position and the absolute position. The apparatus described in Patent Document 1 is merely a device that calculates a tomographic position for an observation target of an endoscope.

  Patent Document 2 describes an ultrasonic endoscope apparatus. This apparatus displays an image obtained by an endoscope and an image obtained by ultrasonic diagnosis in association with each other, and a cross-correlation operation is used for the association. In this apparatus, the endoscope is provided with a CCD camera and an ultrasonic transducer, and three-dimensional ultrasonic data is acquired by manual scanning for pulling the endoscope forward. In addition, this apparatus spatially associates three-dimensional surface shape data obtained by an endoscope with three-dimensional surface shape data obtained by ultrasonic diagnosis. It does not provide support images for endoscope positioning.

  An object of the present invention is to provide an image capable of supporting an endoscope positioning operation.

  Another object of the present invention is to provide an image reflecting the structure on the tissue surface as well as the structure inside the tissue.

  The system according to the present invention includes an endoscopic unit provided with a sensor for observing a local tissue in a visual field existing ahead, and an endoscopic image forming unit that forms an endoscopic image based on a signal from the sensor An ultrasonic probe for acquiring volume data by transmitting and receiving ultrasonic waves to and from a three-dimensional space including the local tissue, and an ultrasound representing a wide-area tissue existing in front of the sensor based on the volume data. An ultrasonic image forming unit that forms a sound image, an image combining unit that forms an image by combining the ultrasonic image representing the wide-area tissue and the endoscopic image representing the local tissue while aligning each other; including.

  According to the above configuration, a composite image obtained by combining the endoscopic image and the ultrasonic image is formed, and is preferably displayed on the screen. The endoscopic image may be an optical observation image similar to the conventional one, or may be an image after processing or correcting it for image synthesis. An ultrasound image is an image of a wide-area tissue in front of the sensor. By synthesizing an endoscopic image against such an ultrasound image, it is easy to grasp the relationship between the local tissue and the surrounding tissue. It becomes. The ultrasonic image is desirably a three-dimensional partial region projected onto a two-dimensional image, and may be, for example, a volume rendering image, an integrated image, or the like. It is desirable that the viewpoint for forming an endoscopic image and the viewpoint for forming an ultrasonic image are approximately close to each other. It is also desirable that the scales of the two images be close to each other. In any case, if some form of surrounding tissue is reflected around the optically observable region, even if some distortion or discontinuity occurs between the two images, It becomes possible to support the operation.

  Preferably, the image synthesizing unit forms the synthesized image by superimposing or fitting the endoscopic image on a part of the ultrasonic image, and the synthesized image includes the endoscopic unit and the ultrasonic probe. It is updated in real time as the relative positional relationship changes. When the superposition method is adopted, the endoscopic image is superimposed on the entire ultrasonic image representing the wide-area tissue. In this case, the endoscopic image may be an overwritten image, and the endoscopic image may be made transparent so that the ultrasonic image portion as the background can be seen through. When the fitting method is adopted, a portion corresponding to the endoscopic image in the ultrasonic image is cut out (perforated imaging), and the endoscopic image is fitted into the cut out portion. In that case, it is desirable to measure the size of the endoscopic image and determine the cutout size of the ultrasonic image based on the result.

  If the composite image is updated in real time, the endoscopic part can be easily positioned. That is, since there is an ultrasonic image portion around the endoscopic image, it is easy to determine in which direction the endoscopic portion should be moved. If the field of view is actually moved, the content of the endoscopic image is updated, so that the diseased part can be easily detected. The movement of the visual field of the endoscopic part is generally performed by an extracorporeal operation. As the visual field moves, the ultrasonic image may be fixed and the endoscopic image may be moved relative to the ultrasonic image, or the endoscopic image may be fixed and the ultrasonic image You may make it move relatively. Or it is also possible to employ | adopt the aspect which both move. In any case, it is desirable to form an appropriate update method from the viewpoint of operation support.

  Preferably, the ultrasonic image forming unit is a specifying unit that specifies an observation origin in the sensor and a visual field direction forward therefrom, and a unit that processes the volume data, the visual field based on the observation origin. Partial volume data processing means for identifying the processing start surface that intersects the direction and processing the partial volume data existing ahead of the processing start surface to form the ultrasonic image. According to this configuration, the observation origin in the sensor is first specified. That is, the spatial coordinates of the endoscopic viewpoint are specified. Further, the viewing direction from the observation origin to the front is specified. The viewing direction preferably coincides with the central axis in the stereoscopic viewing area. That is, partial volume data existing forward from the observation origin is specified so that the state observed from the sensor is reproduced on the ultrasonic image, and an ultrasonic image is formed using the partial volume data. An ultrasonic probe may be brought into contact with the body surface and volume data may be acquired therefrom, or volume data may be acquired from the body using an intracavity probe.

  Preferably, the partial volume data processing means sets a plurality of lines of sight extending forward from the processing start surface, and performs a rendering operation on each line of sight. According to this configuration, the partial volume data is preferably projected in accordance with the parallel projection method. However, a line-of-sight group that radiates from the sensor or its periphery may be set. It is desirable to set the line-of-sight group arrangement according to the visual field characteristics of the sensor (and the optical system). As a rendering method, a volume rendering method, an integral projection method, a maximum value projection method, a minimum value projection method, and the like can be considered. In any case, it is desirable to select a rendering method that can reflect a three-dimensional space in an image.

  Preferably, the ultrasonic image is an image that can represent both a surface blood vessel that runs on the surface of a wide area tissue existing in front of the sensor and an internal blood vessel that runs inside the wide area tissue. If the internal blood vessels are imaged, safety in surgery can be improved.

  Desirably, it includes means for acquiring position information for specifying a positional relationship between the endoscopic part and the ultrasonic probe, and the observation origin and the visual field direction are specified based on the position information. The position information can be acquired using a positioning system using a magnetic field, a positioning system using infrared light, a positioning system using radio waves, and the like. If the position to which the ultrasonic probe is applied is determined and the insertion position of the endoscopic part is determined to some extent, it is possible to manually position and combine the two images, but in that case real-time characteristics Is damaged. Therefore, it is desirable that the relative positional relationship between the ultrasonic probe and the endoscopic unit can be measured in real time.

  Preferably, the means for acquiring the position information includes a first position detection unit that measures the position of the ultrasonic probe and a second position detection unit that measures the position of the endoscopic unit. Here, when each unit is configured by a positioning system using a magnetic field, each unit is configured by a magnetic field generator and a magnetic sensor set. When each unit is composed of a positioning system using infrared rays, each unit is composed of an infrared transducer and an infrared reflector. A positioning device may be provided at the distal end of the endoscopic unit, and if the endoscopic unit is a rod-like body or a shape that does not change so much, a positioning device is provided on the external part of the endoscopic unit. Is possible. If matching processing of both images is performed using image processing such as correlation calculation, the positioning accuracy of both images can be improved by image processing even if the measurement accuracy of the position information is not high.

  Preferably, the image composition unit superimposes the endoscopic image on a part of the ultrasonic image as a background. Preferably, the ultrasonic image is a monochromatic image and the endoscopic image is a color image. Preferably, the image synthesizing unit repeats a correlation calculation while sequentially changing a relative positional relationship between the endoscopic image and the ultrasonic image, thereby performing the endoscopic image and the ultrasonic image. Search for the appropriate positional relationship.

  Preferably, the image composition unit applies a binarization process to the endoscopic image, forms a binarized endoscopic image, and applies a binarization process to the ultrasonic image. Means for forming a binarized ultrasonic image; and means for executing a correlation operation while sequentially changing a relative positional relationship between the binarized endoscopic image and the binarized ultrasonic image And means for specifying the appropriate positional relationship based on the result of the correlation calculation. Correlation calculation is facilitated by binarizing each image even if it is a different type of image.

  Preferably, in the sequential change of the relative positional relationship, the relative scale ratio is variable, the relative translation and the relative rotation are performed between the binarized endoscopic image and the binarized ultrasonic image. At least one of the movements is performed. Particularly preferably, all of them are executed in multiple.

  Preferably, the tissue to be observed is a placenta, the endoscopic image includes an image of blood vessels running on the placenta, and the ultrasound image includes an image of blood vessels running on the placenta and the An image of a blood vessel traveling in the placenta is included, and the endoscopic image and the ultrasound image are updated in real time at least with the movement of the endoscopic unit, and the endoscopic unit is observed under observation of the composite image. The inserted surgical instrument is operated.

  An ultrasonic diagnostic apparatus according to the present invention includes an endoscopic unit provided with a sensor for observing a local tissue in a visual field existing in front, and an endoscopic image that forms an endoscopic image based on a signal from the sensor. An ultrasonic diagnostic apparatus connected to an endoscopic device including an image forming unit, and acquiring volume data by transmitting and receiving ultrasonic waves to and from a three-dimensional space including the local tissue An ultrasonic image forming unit that forms an ultrasonic image representing a wide-area tissue existing in front of the sensor based on the volume data, an ultrasonic image representing the wide-area tissue, and an endoscopic image representing the local tissue And an image composition unit that forms a composite image by aligning them with each other.

  As described above, according to the present invention, it is possible to provide an image that can support the positioning operation of the endoscope. Alternatively, an image reflecting the structure inside the tissue as well as the structure on the tissue surface can be provided.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the drawings.

  FIG. 1 shows a preferred embodiment of a medical image processing system according to the present invention, and FIG. 1 is a block diagram showing the overall configuration thereof. The system shown in FIG. 1 is a system that combines endoscopic treatment and ultrasound diagnosis, and the target tissue in this embodiment is the placenta in the uterus. Of course, the system according to the present invention can be used for other tissues.

  In FIG. 1, the system according to the present embodiment is roughly composed of an ultrasonic diagnostic apparatus 10 and an endoscopic apparatus 12. However, it is also possible to entrust the image processing function to an external computer or the like. In this case, the present system is configured by an ultrasonic diagnostic apparatus, an endoscopic apparatus, and a computer. Various configurations are conceivable for realizing the system according to the present embodiment.

  The ultrasonic diagnostic apparatus 10 will be described. The probe 14 is a transmitter / receiver that transmits / receives an ultrasonic wave. In this embodiment, a 3D probe that transmits / receives an ultrasonic wave to / from a three-dimensional space in a living body is used. Specifically, the probe 14 includes a probe head, a probe cable, a probe connector, and the like. An array transducer 16 is provided in the probe head. The array transducer 16 is a 2D array transducer in the present embodiment, and an ultrasonic beam is formed by the 2D array transducer 16, and the ultrasonic beam is scanned two-dimensionally to form a three-dimensional data capturing space. The Volume data is acquired from within the three-dimensional space. Of course, a three-dimensional space may be formed by mechanically scanning the 1D array transducer.

Reference numeral 18 denotes a positioning system. The positioning system 18 includes a movable device 20, a fixed device 22, and a coordinate data calculation unit 24. A positioning system 48 described later has a similar configuration. For example, when a method of performing positioning using a magnetic field is adopted, the fixed device 22 is a magnetic field generator, and the movable device 20 is configured by a magnetic sensor unit. The output of the magnetic sensor unit is sent to the coordinate data calculation unit 24, and the coordinate data calculation unit 24 measures the spatial coordinates of the movable device 20 with the fixed device 22 as the coordinate origin. If the positioning system 18 uses infrared rays, the fixed side device 22 is an infrared transducer, and the movable side device 20 is an infrared reflector. Various other methods can be adopted. In any case, the spatial coordinates of the probe 14 (specifically, the probe head) are measured. When a configuration in which the probe head is supported using a robot arm or the like is employed, it is also possible to obtain the three-dimensional coordinates of the probe head from the output of the encoder provided in each arm. The coordinate data calculation unit 24 outputs coordinate data (X _P , Y _P , Z _P , Xθ, Yθ, Zθ) to an ultrasonic image forming unit 32 described later.

  The transmission / reception unit 26 includes a transmission beam former as a transmission unit and a reception beam former as a reception unit. At the time of transmission, a plurality of transmission signals are supplied in parallel from the transmission / reception unit 26 to the array transducer 16. As a result, a transmission beam is formed in the array transducer 16. At the time of reception, the reflected wave from the living body is received by the array transducer 16, and a plurality of received signals are output from the array transducer 16 to the transmission / reception unit 26 in parallel. In the transmission / reception unit 26, a phasing addition process is applied to a plurality of reception signals, thereby obtaining a reception signal after phasing addition, ie, beam data.

  The beam data is output to the signal processing unit 28. The signal processing unit 28 includes various circuits such as a detection circuit and a logarithmic compression circuit, and executes a predetermined signal on the beam data. The processed beam data is stored in the 3D memory 30. Coordinate transformation is applied at the time of writing, and echo data (voxel data) constituting each beam data is stored at the corresponding address. Of course, coordinate transformation may be applied at the time of reading. The coordinate conversion generally means conversion from a transmission / reception wave coordinate system to a memory space coordinate system. As a result, volume data is stored in the 3D memory 30.

  In the present embodiment, a three-dimensional eco data capturing space, that is, a three-dimensional space, is formed so as to include a local tissue observed by the endoscopic device 12, and conversely, at such an appropriate position. The 3D probe 14 is positioned so that a three-dimensional space is formed. Accordingly, the 3D memory 30 stores volume data corresponding to a wide area organization including a local tissue that is an object of endoscopic imaging.

The ultrasonic image forming unit 32 forms an ultrasonic image as a projection image by performing rendering processing based on partial data (partial volume data) in the volume data. In that case, data processing is executed so that an ultrasonic image is formed from the same direction as the observation point, that is, the viewpoint in the endoscopic image. At that time, the coordinate data output from the coordinate data calculation unit 24 is referred to, and the coordinate data (x _c , y _c , z _c , xθ, yθ output from the coordinate data calculation unit 54 described later). , zθ). That is, the three-dimensional coordinates of the probe head and the three-dimensional coordinates of the endoscope are referred to, and based on the spatial positional relationship between the two, partial volume data is identified, that is, the viewpoint and the rendering direction are identified. Projection processing is executed using. In that case, a known spatial positional relationship between the two positioning systems 18 and 48 is used as necessary, and the positions of the movable devices 20 and 50 and the points to be subjected to coordinate measurement are also determined. Information such as relationships is referenced.

  The image synthesizing unit 34 is a module that synthesizes an endoscopic image described later on an ultrasonic image. As a synthesis method, a superposition method and a fitting method are conceivable. In the superposition method, the endoscopic image is partially overlapped on the ultrasonic image. When the fitting method is employed, the region where the endoscopic image is fitted is removed from the ultrasonic image, that is, a hole is formed, and the endoscopic image is fitted into the vacant portion. Of course, various conventionally known methods can be employed as the image composition method. In the present embodiment, in the synthesis of two images, the correlation calculation is sequentially performed while changing the relative positional relationship between the two in a stepwise manner. The relationship is certified. If the two images are combined under such an appropriate positional relationship, the positional deviation between the images can be minimized, and the distortion between the images can be reduced. However, such correlation calculation only needs to be executed as necessary, and a certain effect can be obtained by simply superimposing two images.

  As described above, when a composite image is generated in the image composition unit 34, the image data is sent to the display unit 36, and the composite image is displayed on the screen of the display unit 36. The composite image is a new image in which an optical image and an acoustic image are combined. Specifically, an ultrasonic image is displayed around an optical image similar to the conventional one, and was not visible in the past. Since it is possible to recognize the structure and state of the peripheral part through an ultrasonic image, it is possible to provide support for endoscopic operation and to quickly detect the affected part. The endoscopic image can be processed and modified prior to image synthesis, and this is the same for the ultrasonic image. For example, edge enhancement processing, contrast enhancement processing, or the like may be performed.

  The main control unit 38 performs operation control of each component included in the ultrasonic diagnostic apparatus 10, and the main control unit 38 is specifically configured by a CPU and an operation program. An operation panel 40 is connected to the main control unit 38, and the user can set operating conditions and input parameters using the operation panel 40. Incidentally, the ultrasonic image forming unit 32 and the image synthesizing unit 34 can be realized as software functions, and such program processing can be performed in the ultrasonic diagnostic apparatus 10, but volume data is stored. It is also possible to transfer to an external PC and perform image processing on the external PC. However, it is desirable that the composite image is updated in real time in support of the endoscope operation, and it is desirable to construct a system that can realize such real time performance. In the present embodiment, the endoscopic device 12 is combined with the ultrasonic diagnostic apparatus 10, that is, the image data from the endoscopic apparatus 12 is used in the ultrasonic diagnostic apparatus 10 and displayed in real time. An endoscopic image that is also displayed in real time on the ultrasonic image is synthesized.

  Next, the endoscopic device 12 will be described. Reference numeral 42 denotes an endoscope. The endoscope 42 includes a part inserted into the body and a part positioned outside the body. An image sensor 44, a light emitter 46, and the like are provided in the body part. The image sensor 44 is a CCD camera, for example. Of course, other image acquisition devices may be provided. The light emitter 46 is a device for illuminating the front when capturing an image. However, the light emitter 46 can be omitted by using, for example, a high-sensitivity infrared sensor. A plurality of channels (not shown) are provided in the endoscope 42, and a surgical instrument is inserted using any of the channels to perform a treatment on a blood vessel or the like running on the placenta surface. It is possible.

  As described above, the positioning system 48 includes the movable device 50, the fixed device 52, and the coordinate data calculation unit 54. In the present embodiment, the movable device 50 is disposed on the external part of the endoscope 42, but the device 50 may of course be disposed on the internal part. In any case, it is desirable that the observation origin, that is, the coordinates of the viewpoint in the image sensor 44 is measured directly or indirectly. The coordinate data output from the coordinate data calculation unit 54 is input to the ultrasonic image forming unit 32 as described above, and the ultrasonic image processing described above is executed using such information.

  An endoscopic image forming unit 47 is provided in the endoscopic device main body 45, and the endoscopic image forming unit 47 forms an endoscopic image as an optical image based on a signal output from the image sensor 44. The image data is output to the image composition unit 34 as described above.

  A three-dimensional space 60 is shown in FIG. This three-dimensional space 60 is an area where ultrasonic waves are transmitted and received, and corresponds to the volume data described above. The three-dimensional space 60 is configured by electronic scanning of an ultrasonic beam in the probe head 56. Although a cubic three-dimensional space 60 is shown in FIG. 2, the shape is not limited to a cube. For example, in the method of mechanically swinging and scanning a convex 1D array transducer, a pyramid three-dimensional space is formed.

  The three-dimensional space 60 includes a part of the uterus in this embodiment, and a part of the placenta 62 is included therein as shown. Reference numeral 62A represents the surface of the placenta, and a plurality of blood vessels 64 run on the surface (surface layer) 62A.

  The endoscope 58 is configured as a hard rod-like member in the present embodiment, and the distal end portion of the endoscope 58 is inserted into the uterus through a hole formed in the abdomen. An indirect portion having flexibility may be provided at the distal end portion of the endoscope 58 or other portion. The above-described image sensor is provided at the distal end portion of the endoscope 58, and a visual field range by the image sensor is indicated by reference numeral 66. As shown in the figure, the visual field range 66 is a very small area on the placenta surface 62A, and it is not easy to detect the diseased part through the visual field range. That is, there is a need to obtain information about the state of the tissue existing around the visual field range 66. Therefore, in the present embodiment, as described above, a combination of an endoscopic image and an ultrasonic image is realized.

  An example of creating an ultrasound image will be described with reference to FIGS. In FIG. 3, in this embodiment, first, an observation origin 58A corresponding to a viewpoint on the image sensor, that is, a center point, in the endoscope 58 is specified. In order to perform the identification, the above-described positioning system is provided. That is, even when the endoscope 58 is operated, the spatial position of the observation origin 58A can be specified in each state. However, even if the position cannot be specified precisely, it is possible to obtain a good matching state between the endoscopic image and the ultrasonic image as a result of the subsequent image correlation processing. When the observation origin 58A is specified, the reference direction extending from the front of the visual field, that is, the visual field direction 70 is specified. The visual field direction 70 corresponds to the central axis of the visual field range 66, or corresponds to the axial direction of the endoscope 58.

  Next, as shown in FIG. 4, a processing start surface 72 is set with reference to the observation origin 58A. Specifically, the processing start surface 72 is set on the three-dimensional storage space of the 3D memory. In the present embodiment, the processing start surface 72 is a surface that passes through the observation origin 58 </ b> A and is orthogonal to the reference direction 72. Although the size of the processing start surface 72 can be arbitrarily set, it is desirable that the size is determined so as to cover at least the periphery of the visual field range 66, and it is particularly desirable that the processing start surface 72 passes through the entire volume data. It is determined as a range having a size of. The processing start surface 72 may be set not at the surface passing through the observation origin 58A but at a position ahead of it. That is, in this embodiment, there is amniotic fluid between the placenta surface and the image sensor, and this amniotic fluid is a part that is hardly reflected in the ultrasonic image, so the processing start surface is set forward as necessary. It is possible to do. Of course, it is also possible to set the processing start surface behind.

  Next, as shown in FIG. 5, a line-of-sight (ray) group 74 is set as a direction orthogonal to the processing start surface 72 as a reference. The line-of-sight group 74 includes a plurality of lines of sight 76. In the present embodiment, the plurality of lines of sight 76 are parallel to each other, but the plurality of lines of sight 76 may be set in a non-parallel relationship. For example, a plurality of lines of sight 76 may be set radially from the observation origin 58A. Then, rendering is performed for each line of sight, that is, a pixel value is determined for each line of sight. By mapping these pixel values on a two-dimensional plane, an ultrasonic image can be constructed. An example of the rendering method is a cumulative projection method, and it is particularly desirable to apply the volume rendering method. In the present embodiment, rendering conditions are determined so that not only the surface layer portion of the placenta but also the inside of the placenta is reflected in the ultrasound image.

  An ultrasonic image formed as a result of such rendering processing is schematically shown in FIG. In FIG. 6, a wavy line circle 84 represents the field of view range by the endoscope. It can be understood that the periphery of the range is widely imaged. There is a blood vessel group 78, and the blood vessel group 78 includes a blood vessel 80 running on the surface layer and a blood vessel 82 running inside. Of course, it is possible to freely change the rendering depth by changing the rendering conditions.

  As described above, when an ultrasound image is configured, the ultrasound image and the endoscopic image are combined, but in this embodiment, correlation calculation is performed between the two images as described above. Has been. That is, the matching process is performed so that the two images are spatially aligned. For this purpose, each image is binarized.

  In FIG. 7A, an endoscopic image 86 is shown. The image is a circular image corresponding to the visual field range. A blood vessel 88 appears inside thereof. A binarization process is performed on this, and a binarized image 90 generated thereby is shown in FIG. 7B. In the binarization process, 1 (or 0) is given to the blood vessel part, and 0 (or 1) is given to the other tissue parts.

  On the other hand, binarization processing is also performed on the ultrasonic image. Specifically, a binarization process is performed on the ultrasound image 76 shown in FIG. 6 using a threshold value for discriminating blood vessels and tissues, and the binary image shown in FIG. This is a converted image 96. A wavy line circle 84 represents the visual field range of the endoscopic image. In the present embodiment, correlation calculation is performed while the positional relationship between the binarized image 96 shown in FIG. 8 and the binarized image 90 shown in FIG. It is repeatedly executed, and it is recognized as an appropriate positional relationship when the best correlation value is obtained. Then, the two ultrasonic images (image before binarization) and the endoscopic image (image before binarization) that are in the proper positional relationship are synthesized, thereby generating a synthesized image. An example is shown in FIG. The composite image 100 is an image obtained by superimposing the endoscopic image 86 on the ultrasonic image 76. In this embodiment, the ultrasonic image 76 is a black and white image, and the endoscopic image 86 is a color image. Since the endoscopic image 86, which is an optical image, is generally clearer, in the synthesized image 100, a state in which the tissue clearly emerges with respect to the background in the circular area of interest is manifested. become. If the endoscope is moved, the visual field range is moved, so that the content of the endoscopic image 86 is updated in real time. In this case, the position of the endoscopic image 86 may be changed on the ultrasonic image 76, or the ultrasonic image 76 is moved or the content thereof is moved in a state where the screen display position of the endoscopic image 86 is fixed. You may make it update. The size of the endoscopic image may be dynamically changed according to the distance by measuring the distance between the image sensor and the tissue. In the present embodiment, even if discontinuity or distortion occurs between two images, an image representing a tissue structure can be displayed around the endoscopic image 86 with a certain degree of certainty. There is an advantage that it is possible to support, and the identification of the affected part can be facilitated. That is, what is actually focused on is the endoscopic image 86, and if the visual field range is moved, a clear part can be moved, and the image existing in the vicinity is only for assistance. Therefore, even if there is some misalignment between the two images, it is possible to improve convenience compared to the conventional case.

  Next, an example of the operation of the system will be described with reference to FIGS. FIG. 10 shows an operation example of the ultrasonic image forming unit 32 shown in FIG. 11, and FIG. 11 shows an operation example of the image composition unit 34 shown in FIG.

  First, referring to FIG. 10, in S101, the observation origin and the reference direction are calculated as shown in FIG. In this case, coordinate data obtained by a positioning system provided in the endoscope is used. Of course, if the observation direction or the observation origin of the endoscope is known, there is no need to use a positioning system. In S102, the processing start surface is specified as shown in FIG. In S103, as shown in FIG. 5, a plurality of lines of sight extending from the processing start surface are set, and by performing a rendering operation along each line of sight (ray), processing for the partial volume data is executed. An ultrasonic image is formed as a rendering image reflecting the data. An example of the ultrasonic image is as shown in FIG.

  Next, referring to FIG. 11, in S201, the endoscopic image obtained by the endoscopic device is binarized, thereby generating a binarized endoscopic image. Similarly, in S202, the ultrasonic image is binarized, thereby generating a binarized ultrasonic image. In S203, both are temporarily combined while the center of the endoscopic binarized image matches the reference axis point of the ultrasonic binarized image. Here, the reference axis point is a point corresponding to the reference axis in the ultrasonic image, and by aligning the central point of the endoscopic binarized image with this point, the two images are roughly aligned in the three-dimensional space. It is possible to make it. Based on the initial state, the search width in the correlation calculation is set as follows.

  In S204 to S206, the correlation calculation is executed while changing the scale ratio of the image. Specifically, one image (preferably an endoscopic binarized image) is enlarged or reduced, and a correlation value is calculated between the two images in S205. Then, until the correlation value is regarded as the maximum in S206, the correlation calculation while changing the scale ratio of the one image is repeatedly executed. When it is determined in S206 that the correlation value is the maximum, correlation value calculation processing is performed while performing parallel movement in S207 to S209.

  Specifically, in S207, one image is translated in the vertical and horizontal directions, and in S208, a correlation value is calculated in the state after the translation. Then, while repeating the parallel movement, the correlation value calculation is repeatedly executed until the correlation value is regarded as the maximum in S209. Even if it is determined in S209 that the correlation value is the maximum, the processes of S210 to S212 are executed. Specifically, one image is rotated in S210, and a correlation value is calculated in S211. This process is repeatedly executed until it is determined in S212 that the correlation value is maximum. If it is determined in S212 that the correlation value is maximum, S213 is executed.

  In S213, even if it is determined whether the correlation value is saturated or not from the transition so far, if it is not saturated, each step after S204 is repeatedly executed. That is, the best matching state of the two images is searched until the correlation value is saturated while changing the combination of conditions. When it is determined that the correlation value is saturated in S213, the ultrasonic image and the endoscopic image are combined and displayed on the screen in S214. In FIG. 11, if three changes of enlargement / reduction processing, parallel movement processing, and rotation processing are included, one or two of them may be implemented. However, by including the enlargement / reduction processing in particular, even if the distance between the image sensor and the object surface is unknown, there is an advantage that the scale of the two images can be matched to construct a composite image without a sense of incongruity. is there.

  In the present embodiment, since an ultrasonic image in which a three-dimensional region is imaged is used, it is possible to display blood vessels that run inside the tissue, thereby using a surgical instrument. There is an advantage that the safety can be improved even in the case where treatment is performed.

1 is a block diagram showing a preferred embodiment of a medical image processing system according to the present invention. It is a conceptual diagram which shows a three-dimensional space. It is a figure for demonstrating the setting of an observation origin and a reference direction. It is a figure for demonstrating the setting of a process start surface. It is a figure for demonstrating the setting of several eyes | visual_axis. It is a figure which shows an example of an ultrasonic image. It is a figure for demonstrating the binarization process with respect to an endoscopic image. It is a figure which shows the ultrasonic image by which the binarization process was carried out. It is a figure which shows an example of a synthesized image. 3 is a flowchart illustrating processing contents of an ultrasonic image forming unit illustrated in FIG. 1. It is a flowchart which shows the content of the image synthetic | combination part shown in FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Ultrasonic diagnostic apparatus, 12 Endoscopic apparatus, 14 Probe, 18 Positioning system, 32 Ultrasonic image forming part, 34 Image composition part, 42 Endoscope, 44 Image sensor, 45 Endoscopic apparatus main body, 47 Endoscopic image formation Department, 48 positioning system.

Claims (14)

An endoscopic portion provided with a sensor for observing a local tissue in a visual field existing in front;
An endoscopic image forming unit that forms an endoscopic image based on a signal from the sensor;
An ultrasonic probe for acquiring volume data by transmitting and receiving ultrasonic waves to and from a three-dimensional space including the local tissue;
An ultrasonic image forming unit that forms an ultrasonic image representing a wide-area tissue existing in front of the sensor based on the volume data;
An image composition unit that forms a composite image by combining the ultrasonic image representing the wide-area tissue and the endoscopic image representing the local tissue while aligning each other;
A medical image processing system comprising: The system of claim 1, wherein
The image composition unit forms the composite image by superimposing or fitting the endoscopic image on a part of the ultrasonic image,
The medical image processing system, wherein the composite image is updated in real time as the relative positional relationship between the endoscopic unit and the ultrasonic probe is changed. The system according to claim 1 or 2,
The ultrasonic image forming unit includes:
A specifying means for specifying an observation origin in the sensor and a visual field direction forward therefrom;
A means for processing the volume data, wherein a processing start surface that intersects the visual field direction is identified based on the observation origin, and the partial volume data that exists in front of the processing start surface is processed to process the volume data; Partial volume data processing means for forming a sonic image;
A medical image processing system comprising: The system of claim 3, wherein
The medical data processing system, wherein the partial volume data processing unit sets a plurality of lines of sight extending forward from the processing start surface and performs a rendering operation on each line of sight. The system of claim 4, wherein
The ultrasonic image is an image that can represent both a surface blood vessel that runs on the surface of a wide-area tissue existing in front of the sensor and an internal blood vessel that runs inside the wide-area tissue. Image processing system. The system of claim 3, wherein
Means for acquiring position information for specifying a positional relationship between the endoscopic unit and the ultrasonic probe;
The observation origin and the visual field direction are specified based on the position information.
A medical image processing system characterized by that. The system of claim 6, wherein
The means for acquiring the position information includes:
A first position detection unit for measuring the position of the ultrasonic probe;
A second position detection unit for measuring the position of the endoscopic part;
A medical image processing system comprising: The system of claim 1, wherein
The medical image processing system, wherein the image synthesizing unit superimposes the endoscopic image on a part of the ultrasonic image as a background to make it translucent. The system according to any one of claims 1 to 8,
The ultrasound image is a monochromatic image;
The medical image processing system, wherein the endoscopic image is a color image. The system according to any one of claims 1 to 9,
The image synthesizing unit repeats the correlation calculation while sequentially changing the relative positional relationship between the endoscopic image and the ultrasonic image, thereby appropriately adjusting the endoscopic image and the ultrasonic image. A medical image processing system characterized by searching for a positional relationship. The system of claim 10, wherein
The image composition unit
Means for applying a binarization process to the endoscopic image to form a binarized endoscopic image;
Means for applying a binarization process to the ultrasonic image to form a binarized ultrasonic image;
Means for performing a correlation operation while sequentially changing a relative positional relationship between the binarized endoscopic image and the binarized ultrasonic image;
Means for identifying the appropriate positional relationship based on the result of the correlation calculation;
A medical image processing system comprising: The system of claim 11, wherein
In the sequential change of the relative positional relationship, among the binarized endoscopic image and the binarized ultrasonic image, the relative scale ratio is variable, the relative parallel movement, and the relative rotational movement. A medical image processing system, wherein at least one of the following is executed. The system according to any one of claims 1 to 12,
The tissue to be observed is the placenta,
The endoscopic image includes an image of blood vessels running on the placenta,
The ultrasound image includes an image of blood vessels that run on the placenta and an image of blood vessels that run in the placenta,
The endoscopic image and the ultrasound image are updated in real time at least with the movement of the endoscopic unit,
A surgical instrument inserted into the endoscopic part is operated under observation of the composite image.
A medical image processing system characterized by that. An endoscope provided with an endoscopic unit provided with a sensor for observing a local tissue in a visual field existing in front and an endoscopic image forming unit configured to form an endoscopic image based on a signal from the sensor An ultrasonic diagnostic apparatus connected to the apparatus,
An ultrasonic probe for acquiring volume data by transmitting and receiving ultrasonic waves to and from a three-dimensional space including the local tissue;
An ultrasonic image forming unit that forms an ultrasonic image representing a wide-area tissue existing in front of the sensor based on the volume data;
An image composition unit that forms a composite image by combining the ultrasonic image representing the wide-area tissue and the endoscopic image representing the local tissue while aligning each other;
An ultrasonic diagnostic apparatus comprising:

Images