JP2011067486A - Image acquisition method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To acquire a composite image in which a special image with enough intensity is synthesized with an ordinary image and acquire a suitable composite image without shifting of an image in an image acquiring device for acquiring a composite image based on an ordinary image and a special image. <P>SOLUTION: A first ordinary image imaged at a first point of time is acquired, and a second ordinary image imaged at a second point of time later than the first point of time is acquired, a feature amount showing a change in motion from an observed part of the first ordinary image to an observed part of the second ordinary image is acquired, a first special image imaged at a first point of time is corrected based on the motion feature amount, and a corrected third special image and the second ordinary image are synthesized to acquired a composite image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image acquisition method and apparatus for acquiring a composite image on the basis of a normal image acquired by irradiating an observed part with normal light and a special image acquired by irradiating the observed part with special light. is there.

  Conventionally, endoscope systems for observing tissue in a body cavity are widely known, and a normal image is obtained by imaging a portion to be observed in a body cavity by irradiation with white light, and this normal image is displayed on a monitor screen. Electronic endoscope systems have been widely put into practical use.

  Moreover, as an endoscope system as described above, for example, in Patent Document 1, an autofluorescence image emitted from an observed part by irradiation of excitation light is captured together with a normal image to obtain autofluorescence images. There has been proposed a fluorescence endoscope system that displays the above image on a monitor screen.

  In addition, as a fluorescence endoscope system, for example, ICG (Indocyanine Green) is introduced into the body in advance, and the fluorescence image of the blood vessel is detected by irradiating the observation part with excitation light and detecting the fluorescence of ICG in the blood vessel. Some have also been proposed.

  For example, in the fluorescence endoscope system described in Patent Document 1, it is proposed to display a normal image and an autofluorescence image by switching them on a monitor.

JP 2005-204905 A

  However, as in the fluorescence endoscope system described in Patent Document 1, it is possible to compare two images if the normal image and the fluorescent image are switched and displayed, but the normal image and the fluorescent image are simultaneously displayed. Since the comparison cannot be made, it is impossible to accurately determine at which position on the normal image the blood vessel exists. In view of this, it is conceivable to synthesize and display the normal image and the fluorescence image.

  However, when observing a blood vessel image using ICG as described above, ICG in blood has a short time to emit fluorescence with sufficient intensity for observation due to the dynamics of blood flow. Therefore, if the normal image currently captured and the fluorescence image are simply synthesized, the ICG has already flowed, and the ICG fluorescence may not sufficiently appear in the synthesized image. In addition, since the timing of light emission varies depending on a part such as a vein or an artery, a plurality of parts of interest cannot be observed at a time.

  In addition, when synthesizing a normal image and a fluorescent image, if the timings at which these images are captured are different, the form of the observed portion will differ depending on the pulsation, resulting in a deviation in the blood vessel image. A complex image cannot be displayed.

  The present invention has been made in view of the above problems, and in a method and apparatus for acquiring a composite image based on a normal image and a special image, a special image having sufficient strength is combined with the normal image. An object of the present invention is to provide an image acquisition method and apparatus capable of acquiring a composite image and acquiring an appropriate composite image without image shift.

  The image acquisition method of the present invention includes a plurality of normal images captured at a predetermined interval by receiving reflected light reflected from the observed portion by irradiating the observed portion with normal light, and a special light observed portion. A plurality of special images picked up at predetermined intervals by receiving light emitted from the observed portion by irradiation of the first and first images picked up at a first time among a plurality of normal images The first special image captured at the first time point among the normal image or the plurality of special images is acquired, and the second normal image captured at the second time point after the first time point or A second special image captured at a second time point after the first time point is acquired, and a change in movement from the observed portion of the first normal image to the observed portion of the second normal image is obtained. Motion feature amount or movement from the observed portion of the first special image to the observed portion of the second special image A motion feature amount indicating a change is acquired, and based on the acquired motion feature amount, the first special image captured at the first time point of the plurality of special images is corrected to obtain a third special image. The acquired third special image and the second normal image are combined to acquire a combined image.

  The image acquisition apparatus of the present invention receives a light irradiation unit that irradiates the observed part with normal light and special light, and reflected light reflected from the observed part by irradiation of the normal light, and receives a plurality of normal images in a predetermined manner. Among the plurality of normal images acquired by the imaging unit and the imaging unit that captures a plurality of special images at a predetermined interval by receiving light emitted from the observed portion by irradiation of the special light while imaging at intervals A first normal image captured at least at a first time point of the first normal image or a first special image captured at a first time point among a plurality of special images, and a first time point A second normal image captured at a second time point after the second time point or a second special image captured at a second time point after the first time point is obtained, and the first normal image is captured. The movement of the second normal image from the observation unit to the observation unit A motion feature amount acquisition unit for acquiring a motion feature amount indicating a change in motion or a motion feature amount indicating a change in motion from the observed portion of the first special image to the observed portion of the second special image, and a motion acquisition unit. A motion correction unit that corrects the first special image captured at a first time point out of the plurality of special images based on the acquired motion feature amount, and acquires a third special image; and a motion correction unit And a synthesized image obtaining unit that obtains a synthesized image by synthesizing the third special image obtained by the above and the second normal image.

  In the image acquisition device of the present invention, at least one special among a display unit that sequentially displays a plurality of special images captured by the imaging unit in time series and a plurality of special images that are sequentially displayed on the display unit. A selection signal receiving unit that receives a signal for selecting an image as the first special image can be provided.

  In addition, a region of interest reception unit that receives a setting of a region of interest in a predetermined normal image or a predetermined special image, and a pixel signal in each region of interest of a plurality of special images corresponding to the region of interest set in the predetermined normal image or A past image selection unit that selects the first special image based on pixel signals in each region of interest of the plurality of special images corresponding to the region of interest set in the predetermined special image can be provided.

  In addition, a display unit that simultaneously displays a plurality of special images captured by the imaging unit, and a signal that selects at least one special image among the plurality of special images simultaneously displayed on the display unit as a first special image A selection signal receiving unit may be provided.

  In addition, a tube image extraction unit that extracts an image representing the tube portion included in the observed portion from the third special image is further provided, and the composite image acquisition unit is an image representing the tube portion extracted by the tube image extraction unit. The synthesized image can be obtained by synthesizing the second normal image.

  Further, the composite image acquisition unit may acquire the composite image by combining the third special image, the second special image, and the second normal image.

  A plurality of first special images can be used.

  According to the image acquisition method and apparatus of the present invention, the first normal image or the first special image captured at the first time point is acquired, and the image is captured at the second time point after the first time point. The obtained second normal image or the second special image is acquired, and the motion feature amount or the first characteristic indicating the change in motion from the observed portion of the first normal image to the observed portion of the second normal image A motion feature amount indicating a change in motion from the observed portion of the special image to the observed portion of the second special image is acquired, and the first image captured at the first time point is acquired based on the acquired motion feature amount. The third special image is acquired by correcting the special image of the image, and the composite image is acquired by combining the acquired third special image and the second normal image. A composite image obtained by compositing an image with a normal image can be acquired, and there is no image shift. It is possible to obtain the appropriate synthesized image.

  That is, for example, when observing a blood vessel image using ICG, the first special image at the first time point when the ICG passes through the observed portion is acquired, and motion correction is performed on the first special image. Since the applied third special image is synthesized with the second normal image captured at the second time point after the first time point, the special image with sufficient ICG emission intensity is obtained as the normal image. It is possible to acquire a composite image synthesized in the above, and it is possible to acquire an appropriate composite image without any deviation in the blood vessel image.

  In the image acquisition device of the present invention, a plurality of special images picked up by the image pickup unit are sequentially displayed in time series, and at least one of the plurality of special images displayed in order is displayed as the first special image. When selecting as an image, the operator can easily select a desired special image to be synthesized.

  In addition, the setting of a region of interest in a predetermined normal image or a predetermined special image is accepted, and in a pixel signal or a predetermined special image in each region of interest of a plurality of special images corresponding to the region of interest set in the predetermined normal image When the first special image is selected on the basis of the pixel signal in each region of interest of the plurality of special images corresponding to the set region of interest, the apparatus automatically operates without being aware of the operator. Since an appropriate image to be synthesized can be selected, the operator's trouble can be saved.

  When a plurality of special images captured by the imaging unit are displayed at the same time, and at least one special image among the plurality of special images displayed at the same time is selected as the first special image, A more appropriate image to be combined can be selected while comparing past images.

  In addition, when the second special image is combined with the second normal image to obtain the composite image in addition to the third special image, the light emission state of the special image at the second time point is also reflected. It is possible to acquire the synthesized image.

  When a plurality of first special images are used, for example, as a first special image, an arterial special image and a vein special image having different emission timings are obtained, and these are acquired. Since a synthesized image synthesized with a normal image can be acquired, a plurality of regions of interest can be observed at once.

Schematic configuration diagram of a laparoscopic system using an embodiment of the image acquisition device of the present invention Schematic configuration diagram of hard insertion part Schematic configuration diagram of the imaging unit The block diagram which shows schematic structure of an image processing apparatus and a light source device Block diagram showing schematic configuration of image processing unit Flowchart for explaining the operation of applying motion correction to the past fluorescent image signal and then combining it with the current normal image signal Flowchart for explaining a method for acquiring a motion feature amount Diagram showing an example of feature points Diagram for explaining the operation of the matching calculation process Diagram for explaining motion correction processing The figure which shows an example of the rectangular area | region of the fluorescence image signal after a motion correction process Flowchart for explaining line segment extraction processing using edge detection Diagram showing an example of a composite image The figure for demonstrating the setting of the region of interest on a fluorescence image The figure which shows an example of the display mode of a selection button The figure which shows an example of the display mode of the thumbnail image of the past fluorescence image

  Hereinafter, a laparoscopic system using an embodiment of an image acquisition device of the present invention will be described in detail with reference to the drawings. FIG. 1 is an external view showing a schematic configuration of a laparoscopic system 1 of the present embodiment.

  As shown in FIG. 1, the laparoscopic system 1 of the present embodiment guides the normal light and special light emitted from the light source device 2 and the light source device 2 that emits white normal light and special light simultaneously. Rigid mirror imaging that irradiates the observed part and captures a normal image based on reflected light reflected from the observed part by normal light irradiation and a fluorescent image based on fluorescence emitted from the observed part by special light irradiation An image processing apparatus 3 that performs predetermined processing on the image signal captured by the apparatus 10, the rigid endoscope imaging apparatus 10, and a normal image and a fluorescence image of the observed portion based on a display control signal generated by the image processing apparatus 3 And a monitor 4 for displaying a composite image.

  As shown in FIG. 1, the rigid endoscope imaging apparatus 10 includes a hard insertion portion 30 that is inserted into the abdominal cavity and an imaging unit 20 that captures a normal image and a fluorescence image of the observed portion guided by the hard insertion portion 30. And.

  In addition, as shown in FIG. 2, the rigid endoscope imaging apparatus 10 has a hard insertion portion 30 and an imaging unit 20 that are detachably connected. The hard insertion portion 30 includes a connection member 30a, an insertion member 30b, a cable connection port 30c, and an irradiation window 30d.

  The connection member 30a is provided on one end side 30X of the hard insertion portion 30 (insertion member 30b). For example, the connection member 30a is fitted into an opening 20a formed on the imaging unit 20 side, so that the imaging unit 20 and the hard insertion portion 30 are fitted. Are detachably connected.

  The insertion member 30b is inserted into the abdominal cavity when photographing inside the abdominal cavity, and is formed of a hard material and has, for example, a cylindrical shape with a diameter of approximately 5 mm. A lens group for forming an image of the observed portion is accommodated inside the insertion member 30b, and the normal image and the fluorescent image of the observed portion incident from the other end 30Y pass through the lens group. The light is emitted to the imaging unit 20 side of the one end side 30X.

  A cable connection port 30c is provided on the side surface of the insertion member 30b, and the optical cable LC is mechanically connected to the cable connection port 30c. Thereby, the light source device 2 and the insertion member 30b are optically connected via the optical cable LC.

  The irradiation window 30d is provided on the other end 30Y of the hard insertion portion 30, and irradiates the observed portion with normal light and special light guided by the optical cable LC. The insertion member 30b accommodates a light guide (not shown) for guiding normal light and special light from the cable connection port 30c to the irradiation window 30d. The irradiation window 30d is guided by the light guide. The target part is irradiated with normal light and special light.

  FIG. 3 is a diagram illustrating a schematic configuration of the imaging unit 20. The imaging unit 20 includes a first imaging system that captures a fluorescent image of the observed portion formed by the lens group in the hard insertion portion 30 and generates a fluorescent image signal of the observed portion, and the hard insertion portion 30. And a second imaging system that generates a normal image signal by capturing a normal image of the observed portion formed by the lens group. These imaging systems are divided into two optical axes orthogonal to each other by a dichroic prism 21 having a spectral characteristic that reflects a normal image and transmits a fluorescent image.

  The first imaging system includes a special light cut filter 22 that cuts off the special light reflected at the observed portion and transmitted through the dichroic prism 21, and the dichroic prism 21 and the special light cut filter 22 emitted from the hard insertion portion 30. A first imaging optical system 23 that images the transmitted fluorescent image L4 and a high-sensitivity imaging element 24 that images the fluorescent image L4 imaged by the first imaging optical system 23 are provided.

  The second imaging system is a normal image formed by the second imaging optical system 25 and a second imaging optical system 25 that forms a normal image L3 emitted from the hard insertion portion 30 and reflected by the dichroic prism 21. An image sensor 26 that captures the image L3 is provided.

  The high-sensitivity imaging element 24 detects light in the wavelength band of the fluorescent image L4 with high sensitivity, converts it into a fluorescent image signal, and outputs it. The high sensitivity image sensor 24 is a monochrome image sensor.

  The image sensor 26 detects light in the wavelength band of the normal image, converts it into a normal image signal, and outputs it. On the image pickup surface of the image pickup element 26, color filters of three primary colors red (R), green (G) and blue (B), or cyan (C), magenta (M) and yellow (Y) are arranged in a Bayer array or a honeycomb. It is provided in an array.

  In addition, the imaging unit 20 includes an imaging control unit 27. The imaging control unit 27 performs CDS / AGC (correlated double sampling / automatic gain control) processing and A / D on the fluorescence image signal output from the high-sensitivity imaging device 24 and the normal image signal output from the imaging device 26. A conversion process is performed and output to the image processing apparatus 3 via the cable 5 (see FIG. 1).

  As shown in FIG. 4, the image processing apparatus 3 includes a normal image input controller 31, a fluorescence image input controller 32, an image processing unit 33, a memory 34, a video output unit 35, an operation unit 36, a TG (timing generator) 37, and a CPU 38. It has.

  The normal image input controller 31 and the fluorescence image input controller 32 include a line buffer having a predetermined capacity, and temporarily store the normal image signal and the fluorescence image signal for each frame output from the imaging control unit 27 of the imaging unit 20. To remember. Then, the normal image signal stored in the normal image input controller 31 and the fluorescent image signal stored in the fluorescent image input controller 32 are stored in the memory 34 via the bus.

  The image processing unit 33 receives a normal image signal and a fluorescence image signal for each frame read from the memory 34, performs predetermined image processing on these image signals, and outputs them to the bus. A more specific configuration of the image processing unit 33 is shown in FIG.

  As shown in FIG. 5, the image processing unit 33 performs normal image processing unit 33 a that performs predetermined image processing suitable for the normal image on the input normal image signal, and outputs the normal image signal. Based on the normal image signal, the motion feature amount acquisition unit 33b that acquires the motion feature amount of the observed portion, and the fluorescence image that is output by performing predetermined image processing suitable for the fluorescence image on the input fluorescence image signal A processing unit 33c, a motion correction unit 33d that performs motion correction on the fluorescence image signal captured in the past based on the motion feature amount, and a past fluorescence image signal that has undergone motion correction by the motion correction unit 33d And a blood vessel extraction unit 33e for performing processing for extracting an image signal representing a blood vessel on the current fluorescence image signal, and an image signal representing a blood vessel extracted from the past fluorescence image signal (hereinafter referred to as “past blood vessel fluorescence image signal”). ) And an image signal representing a blood vessel extracted from the current fluorescence image signal (hereinafter referred to as “current blood vessel fluorescence image signal”) and a normal image signal output from the normal image processing unit 33a to synthesize the composite image. And an image composition unit 33f for generating a signal. The processing in each part of the image processing unit 33 will be described in detail later.

  The video output unit 35 receives the normal image signal, the fluorescence image signal, and the composite image signal output from the image processing unit 33 via the bus, performs predetermined processing to generate a display control signal, and displays the display control signal. Is output to the monitor 4.

  The operation unit 36 receives input by the operator such as various operation instructions and control parameters. The TG 37 outputs a driving pulse signal for driving the high-sensitivity imaging device 24, the imaging device 26 of the imaging unit 20, and the LD driver 45 of the light source device 2 described later. The CPU 36 controls the entire apparatus.

  As shown in FIG. 4, the light source device 2 condenses the normal light source 40 that emits normal light (white light) L <b> 1 having a broadband wavelength of about 400 to 700 nm and the normal light L <b> 1 emitted from the normal light source 40. And the normal light L1 collected by the condensing lens 42 is transmitted, the special light L2 described later is reflected, and the normal light L1 and the special light L2 are incident on the incident end of the optical cable LC. And a dichroic mirror 43. For example, a xenon lamp is used as the normal light source 40. A diaphragm 41 is provided between the normal light source 40 and the condenser lens 42, and the amount of the diaphragm is controlled based on a control signal from ALC (Automatic light control).

  The light source device 2 is light in the visible to near-infrared band of 700 nm to 800 nm. For example, when ICG (Indocyanine Green) is used as a fluorescent dye, near-infrared light of 750 to 790 nm is used as the special light L2. The LD light source 44 that emits the light, the LD driver 45 that drives the LD light source 44, the condensing lens 46 that condenses the special light L2 emitted from the LD light source 44, and the special light condensed by the condensing lens 46 And a mirror 47 that reflects L2 toward the dichroic mirror 43.

  As the special light L2, a wavelength in a narrower band than normal light having a wideband wavelength is used. The special light L2 is not limited to light in the above wavelength range, and is determined as appropriate depending on the type of fluorescent dye or the type of living tissue to be autofluorescent.

  The light source device 2 is optically connected to the rigid mirror imaging device 10 via the optical cable LC.

  Next, the operation of the laparoscopic system of this embodiment will be described.

  First, after the hard insertion portion 30 and the cable 5 to which the optical cable LC is connected are attached to the imaging unit 20, the light source device 2, the imaging unit 20, and the image processing device 3 are powered on and driven.

  Next, the hard insertion portion 30 is inserted into the abdominal cavity by the operator, and the distal end of the hard insertion portion 30 is placed in the vicinity of the observed portion.

  Then, the normal light L1 emitted from the normal light source 40 of the light source device 2 is incident on the hard insertion portion 30 via the condenser lens 42, the dichroic mirror 43, and the optical cable LC, and is irradiated from the irradiation window 30d of the hard insertion portion 30. Irradiate the observation part. On the other hand, the special light L2 emitted from the LD light source 44 of the light source device 2 is incident on the hard insertion portion 30 via the condenser lens 46, the mirror 47, the dichroic mirror 43, and the optical cable LC, and the irradiation window of the hard insertion portion 30 From 30d, the observed part is irradiated simultaneously with the normal light. Note that the simultaneous irradiation does not necessarily mean that the irradiation periods are completely the same, and it is sufficient that at least some of the irradiation periods overlap.

  Then, a normal image based on the reflected light reflected from the observed portion by the irradiation of the normal light L1 is captured, and a fluorescent image based on the fluorescence emitted from the observed portion by the irradiation of the special light L2 is simultaneously with the normal image. Imaged. It should be noted that ICG is administered to the observed part in advance, and fluorescence emitted from the ICG is imaged.

  More specifically, when the normal image is captured, the normal image L3 based on the reflected light reflected from the observed portion by the irradiation of the normal light L1 is incident from the tip 30Y of the insertion member 30b, and the inside of the insertion member 30b. Are guided by the lens group and emitted toward the imaging unit 20.

  The normal image L3 incident on the imaging unit 20 is reflected by the dichroic prism 21 in the direction perpendicular to the imaging element 26, and is imaged on the imaging surface of the imaging element 26 by the second imaging optical system 25. 26 sequentially captures images at predetermined intervals. In the present embodiment, it is assumed that a normal image is captured at a frame rate of 30 fps.

  The normal image signal sequentially output from the image sensor 26 is subjected to CDS / AGC (correlated double sampling / automatic gain control) processing and A / D conversion processing in the imaging control unit 27, and then the image is transmitted through the cable 5. The data is sequentially output to the processing device 3.

  On the other hand, when the fluorescent image is picked up, a fluorescent image L4 based on the fluorescence emitted from the observed portion by the special light irradiation enters from the tip 30Y of the insertion member 30b and is guided by the lens group in the insertion member 30b. And emitted toward the imaging unit 20.

  The fluorescent image L4 incident on the imaging unit 20 passes through the dichroic prism 21 and the special light cut filter 22, and then is imaged on the imaging surface of the high-sensitivity imaging device 24 by the first imaging optical system 23, and has high sensitivity. Images are taken at a predetermined interval by the image sensor 24. In the present embodiment, it is assumed that the normal image is captured at a frame rate of 5 to 10 fps.

  The fluorescent image signals sequentially output from the high-sensitivity image sensor 24 are subjected to CDS / AGC (correlated double sampling / automatic gain control) processing and A / D conversion processing in the imaging control unit 27, and then passed through the cable 5. Are sequentially output to the image processing apparatus 3.

  Next, the operation of displaying the normal image, the fluorescence image, and the composite image based on the normal image signal and the fluorescence image signal captured by the imaging unit 20 as described above will be described with reference to FIGS. .

  First, the operation of displaying a normal image and a fluorescent image will be described. The normal image signal input to the image processing device 3 is temporarily stored in the normal image input controller 31 and then stored in the memory 34. The normal image signal for each frame read from the memory 34 is subjected to gradation correction processing and sharpness correction processing in the normal image processing unit 33a of the image processing unit 33, and then sequentially output to the video output unit 35. Is done.

  Then, the video output unit 35 performs a predetermined process on the input normal image signal to generate a display control signal, and sequentially outputs the display control signal for each frame to the monitor 4. The monitor 4 displays a normal image based on the input display control signal.

  The fluorescence image signal input to the image processing device 3 is temporarily stored in the fluorescence image input controller 32 and then stored in the memory 34. Then, the fluorescence image signal for each frame read from the memory 34 is subjected to predetermined image processing in the fluorescence image processing unit 33 c of the image processing unit 33 and then sequentially output to the video output unit 35.

  The video output unit 35 performs a predetermined process on the input fluorescent image signal to generate a display control signal, and sequentially outputs the display control signal for each frame to the monitor 4. The monitor 4 displays a fluorescent image based on the input display control signal.

  Next, the operation of displaying the composite image by combining the normal image signal and the fluorescence image signal will be described.

  Here, in the laparoscopic system of the present embodiment, the fluorescence of ICG flowing in the blood vessel of the observed part is imaged as described above, but the ICG does not stay in the same place of the blood vessel but flows with the blood flow. Therefore, simply synthesizing the normal image signal and the fluorescent image signal that are currently captured is after the ICG has already flowed, and the fluorescence of the ICG sufficiently appears in the current fluorescent image signal. There may not be.

  Therefore, in the laparoscopic system of the present embodiment, not only the current fluorescence image signal but also the past fluorescence imaged at the time when the ICG was flowing in the blood vessel of the observed portion with respect to the current normal image signal. The image signal is also synthesized.

  Further, when the past fluorescent image signal is combined with the current normal image signal, the form of the observed portion when the past fluorescent image signal is imaged and the observed portion when the current normal image signal is imaged. The form may differ depending on the pulsation. In such a case, if the past fluorescence image signal and the current normal image signal are synthesized as they are, a deviation occurs in the blood vessel image and appropriate synthesis is performed. An image cannot be generated.

  Therefore, in the laparoscopic system of the present embodiment, motion correction is performed on the past fluorescence image signal so that the blood vessel image does not shift as described above.

  Hereinafter, the operation of performing motion correction on the past fluorescent image signal and then combining it with the current normal image signal will be described in more detail.

  First, as described above, the memory 34 of the image processing apparatus 3 stores a large number of normal image signals and fluorescent image signals that are imaged in time series at predetermined intervals in the imaging unit 20.

  Then, as shown in FIG. 6, among the many past fluorescence image signals stored in the memory 34, one fluorescence image signal in which the fluorescence of ICG appears to appear appropriately is designated as the first fluorescence image signal IF. (Old) is acquired (S10 in FIG. 6). A method for selecting the first fluorescent image signal IF (old) will be described in detail later.

  On the other hand, among the many past normal image signals stored in the memory 34, one normal image signal captured at the same time as the first fluorescent image signal IF (old) selected in S10 is used as the first normal image. Obtained as a signal IV (old) (S12 in FIG. 6). Note that, as described above, the imaging intervals (frame rates) of the normal image signal and the fluorescence image signal are different from each other, but they are assumed to be in phase. That is, since the fluorescence image signal has a longer imaging interval, it is assumed that the normal image signal is always captured at the timing when the fluorescence image signal is captured. It is assumed that the normal image signal and the fluorescence image signal captured at the same imaging timing are associated with each other and stored in the memory 34.

  Then, the first normal image signal IV (old) and the current normal image signal IV (new) (hereinafter referred to as “second normal image signal IV (new)”) acquired as described above move. It is input to the feature amount acquisition unit 33b. Then, the motion feature amount acquisition unit 33b calculates the motion feature amount of the observed portion based on the first normal image signal IV (old) and the second normal image signal IV (new) (FIG. 6). S14).

  Hereinafter, a method for calculating the motion feature amount will be described in detail with reference to the flowchart of FIG.

  First, as described above, the motion feature quantity acquisition unit 33b acquires the first normal image signal IV (old) and the second normal image signal IV (new) (S30 in FIG. 7). Then, the first and second normal image signals IV (old) and IV (new), which are color image signals, are subjected to gray image processing and converted into gray image signals (S32 in FIG. 7). Next, contrast enhancement processing is performed on the first and second normal image signals IV (old) and IV (new) which are gray image signals (S34 in FIG. 7). The reason why the gray image processing and the contrast enhancement processing are performed in this way is to facilitate feature point extraction in feature point extraction processing described later.

Next, multi-resolution image generation processing is performed on the first and second normal image signals IV (old) and IV (new) on which the contrast enhancement processing has been performed. A normal image signal having a resolution of 1 is generated (S36 in FIG. 7). Hereinafter, normal image signals of various resolutions generated based on the first normal image signal IV (old) are referred to as first multi-resolution image signals IV (old) 1 to _m, and the second normal image signal. The normal image signals having various resolutions generated based on IV (new) are _{defined as} second multi-resolution signals IV (new) 1 to _n . In this way, the multi-resolution image generation process is performed by extracting rough feature points using a coarse-resolution multi-resolution image signal and fine-resolution using a multi-resolution image signal having a fine resolution. This is for extracting feature points.

The first and second multi-resolution image signals IV (old) 1- _m and IV (new) 1- _n generated as described above are subjected to smoothing processing for noise removal. (S38 in FIG. 7).

Next, feature point extraction processing is performed on the first and second multi-resolution image signals IV (old) 1 _-m and IV (new) 1- _n that have been subjected to smoothing processing, and each multi-resolution image is processed. Feature points are extracted from the signal (S40 in FIG. 7). As the feature point extraction process, for example, a so-called corner extraction method using the Hessain method or the Harris method can be used. Alternatively, feature points may be extracted using DOG (Difference of Gaussian) or HOG (Histograms of Oriented Gradients). Since these feature point extraction methods are known, a detailed description thereof will be omitted. FIG. 8 shows an example of a multi-resolution image from which feature points have been extracted by feature point extraction processing.

  Then, a matching calculation process is performed on the feature points extracted for the first normal image signal IV (old) and the feature points extracted for the second normal image signal IV (new). (S42 in FIG. 7). The matching calculation process is a calculation process for detecting a pair of feature points corresponding to the feature points of the first normal image signal IV (old) and the second normal image signal IV (new). Specifically, for example, a Euclidean distance between a predetermined feature point in one normal image signal and all feature points in the other normal image signal is calculated, and two feature points having the minimum Euclidean distance are calculated. What is necessary is just to make it detect as a set of corresponding feature points. FIG. 9 shows a schematic diagram in which feature points corresponding to the first normal image signal IV (old) and the second normal image signal IV (new) are indicated by straight lines.

  Then, based on the result of the matching calculation process, a motion vector indicating the direction from each feature point of the first normal image signal IV (old) to the corresponding feature point of the second normal image signal IV (new). Is calculated as the movement feature amount (S44 in FIG. 7).

  As described above, based on the first normal image signal IV (old) and the second normal image signal IV (new), the motion feature amount of the observed portion is calculated (S14 in FIG. 6).

  Next, the motion feature amount acquired by the motion feature amount acquisition unit 33b is output to the motion correction unit 33d. Then, the motion correction unit 33d performs a motion correction process on the first fluorescent image signal IF (old) based on the input motion feature amount (S16 in FIG. 6). Specifically, first, as shown in FIG. 10, the first fluorescent image signal IF (old) is divided into rectangular regions of a predetermined size, and the first fluorescent image signal IF (old) is compared with the first fluorescent image signal IF (old). Each feature point of one normal image signal IV (old) is associated with its motion vector. Then, the first fluorescent image signal IF (old) is obtained by deforming each rectangular area based on the motion vector of the characteristic point D having the highest evaluation value among the characteristic points D existing in the vicinity of the corner G of each rectangular area. Transform. FIG. 11 shows an example of the rectangular region after deformation and its corner G ′.

  Then, the first fluorescence image signal IF (old) (hereinafter referred to as “third fluorescence image signal IF ′ (old)”) subjected to the motion correction processing obtained as described above and the second fluorescence The image signal IF (new) is input to the blood vessel extraction unit 33e of the image processing unit 33. Then, blood vessel extraction processing is performed on each fluorescent image signal in the blood vessel extraction unit 33e (S18, S20 in FIG. 6).

  The blood vessel extraction process is performed by performing a line segment extraction process. In this embodiment, line segment extraction processing is performed by removing isolated points from edges detected by edge detection and edge detection. As an edge detection method, for example, a Canny method using first-order differentiation can be used. FIG. 12 is a flowchart for explaining line segment extraction processing using edge detection by the Canny method.

  As shown in FIG. 12, first, a filtering process using a DOG (Derivative of Gaussian) filter is performed on each of the second fluorescence image signal IF (new) and the third fluorescence image signal IF ′ (old). (S50 to S54 in FIG. 12). The filter process using the DOG filter is a process in which a Gaussian filter process (smoothing process) for reducing noise and a primary differential filter process in the x and y directions for detecting a density gradient are combined.

  Then, the magnitude and direction of the density gradient are calculated for each of the filtered second fluorescent image signal IF (new) and third fluorescent image signal IF ′ (old) (S56 in FIG. 12). Then, the local maximum point of the concentration gradient is extracted, and other non-maximal points are removed (S58 in FIG. 12).

  Then, the maximum point is compared with a predetermined threshold value, and a maximum point equal to or greater than the predetermined threshold value is detected as an edge (S60 in FIG. 12). Further, a process of removing isolated points that are maximal points and are equal to or greater than a predetermined threshold but do not constitute a continuous edge is performed (S62 in FIG. 12). The isolated point removal process is a process for removing an isolated point that is not suitable as a blood vessel from the edge detection result. Specifically, the isolated point is detected by checking the length of each detected edge.

  Note that the edge detection algorithm is not limited to the above, and the edge detection algorithm uses a LOG (Laplace of Gaussian) filter that combines a Laplacian filter that extracts edges by performing Gaussian filter processing for reducing noise and second-order differentiation processing. Detection may be performed.

  In the present embodiment, blood vessel extraction is performed by performing line segment extraction processing using edge detection. However, the present invention is not limited to this. Any processing, such as processing using the, may be used.

  By performing the blood vessel extraction process as described above, the current blood vessel fluorescent image signal and the past blood vessel fluorescent image signal are obtained for each of the second fluorescent image signal IF (new) and the third fluorescent image signal IF ′ (old). Generated.

  Then, the current blood vessel fluorescence image signal and the past blood vessel fluorescence image signal generated in the blood vessel extraction unit 33e are combined to generate a combined blood vessel fluorescence image signal (S22 in FIG. 6).

  The synthetic blood vessel fluorescence image signal generated in the blood vessel extraction unit 33e and the second normal image signal IV (now) are output to the image synthesis unit 33f, and the image synthesis unit 33f outputs the second normal image signal IV (now). Are combined with each other to generate a composite image signal (S24 in FIG. 6).

  Then, the composite image signal generated by the image composition unit 33 f is output to the video output unit 35.

  Then, the video output unit 35 performs a predetermined process on the input composite image signal to generate a display control signal, and outputs the display control signal to the monitor 4. Then, the monitor 4 displays a composite image based on the input display control signal. FIG. 13 shows an example of a composite image displayed based on the composite image signal.

  As described above, not only the current fluorescence image signal but also the past fluorescence image signal captured at the time when the ICG was flowing in the blood vessel of the observed portion is synthesized with the current normal image signal. By displaying, the blood vessel image can be clearly displayed.

  Next, a method for selecting a past fluorescence image signal to be combined with the current normal image signal will be described.

  As one method of selecting past fluorescent image signals, for example, when the current fluorescent image is sequentially displayed on the monitor 4 in time series, a region of interest R is set on the fluorescent image as shown in FIG. Then, a fluorescent image signal having the maximum pixel value information in the region of interest among a large number of past fluorescent image signals may be selected as the past fluorescent image signal IF (old) to be synthesized. The region of interest R may be set by, for example, accepting a setting input from the operation unit 36 by the CPU 38 for a region that the operator thinks that the ICG fluorescence is strong. Further, as the pixel value information, for example, the maximum value or the average value of the pixel values in the region of interest may be used. Also, instead of setting the region of interest directly on the fluorescent image, the region of interest is set on the normal image, and the region of interest on the fluorescent image is set indirectly by associating the region of interest on the fluorescent image. You may do it.

  The region of interest may be set on a normal image that is sequentially displayed on the monitor 4 in time series.

  Further, as shown in FIG. 15, when the current fluorescent image is sequentially displayed on the monitor 4 in time series, a selection button S for selecting the fluorescent image displayed at a predetermined timing is displayed. When the CPU 38 receives a signal that the operator presses the selection button S with a pointing device such as a mouse, the fluorescent image signal displayed at that timing is selected as the past fluorescent image signal IF (old) to be synthesized. May be.

  In addition, as shown in FIG. 16, the current fluorescent images are sequentially displayed on the monitor 4 in time series, and a large number of past fluorescent images P1 to P6 are also simultaneously displayed as thumbnail images, and these past fluorescent images P1 to P6 are displayed. The operator selects a desired fluorescence image from the frame frame F or the like from P6, and the selection signal is received by the CPU 38, whereby the fluorescence image signal selected by the operator is converted into the past fluorescence image signal IF (old) to be synthesized. ).

  In the above embodiment, the movement feature amount of the observed portion is acquired based on the first normal image signal IV (old) and the second normal image signal IV (new). This is because the normal image signal is more sensitive and has a higher frame rate, so the follow-up to the motion is higher, and the normal image signal is less blurred, but the movement of the observed part is If it is not so intense, the motion feature amount may be acquired based on the first fluorescent image signal IF (old) and the second fluorescent image signal IF (new). The acquisition method is almost the same as that of the normal image signal described above.

  In the above embodiment, only one past fluorescence image signal IF (old) to be synthesized is selected. For example, the timing at which the ICG passes between the artery and the vein is different. Since the timing of strengthening is different, the past fluorescence image signal imaged at the timing when the artery is strongly emitting fluorescence and the past fluorescence image signal imaged at the timing when the vein is strongly emitting fluorescence are used. After selecting them and applying motion correction processing to them, they may be combined with the current normal image signal IV (new). In addition, two or more past fluorescent image signals may be synthesized.

  In the above embodiment, a xenon lamp is used as a normal light source. However, the present invention is not limited to this, and a high-intensity white light source using a GaN-based semiconductor laser (trade name: Micro White, Nichia Corporation) May be used. This high-intensity white light source guides light emitted from a semiconductor laser having a wavelength of 445 nm to an optical fiber using an optical lens, and emits white light having a total luminous flux of 501 nm from the other end face of the optical fiber coated with a phosphor material. Is to be released.

  In the above embodiment, the LD light source is used as the special light source. However, the present invention is not limited to this, and an excitation filter is added to the xenon lamp so that normal light and special light are irradiated in time series. Good.

  In the above embodiment, the fluorescent image is picked up by the first image pickup system. However, the present invention is not limited to this, and an image based on the light absorption characteristic of the observed part by special light irradiation to the observed part is picked up. You may make it do.

  In the above-described embodiment, the blood vessel image is extracted. However, the present invention is not limited to this, and for example, an image representing other duct parts such as lymphatic vessels and bile ducts may be extracted.

  Moreover, although the said embodiment applies the image acquisition apparatus of this invention to a laparoscopic system, it is not restricted to this, For example, even if applied to the other endoscope system which has a flexible endoscope apparatus. Good. Further, the present invention is not limited to an endoscope system, and may be applied to a so-called video camera type medical image capturing apparatus that does not include an insertion portion that is inserted into the body.

DESCRIPTION OF SYMBOLS 1 Laparoscope system 2 Light source device 3 Image processing device 4 Monitor 10 Rigid endoscope imaging device 20 Imaging unit 30 Hard insertion part 33 Image processing part 33a Normal image processing part 33b Motion feature-value acquisition part 33c Fluorescence image processing part 33d Correction | amendment part 33e Blood vessel Extraction unit 33f Image composition unit 40 Normal light source 44 LD light source

Claims (8)

A plurality of normal images received at a predetermined interval by receiving reflected light reflected from the observed portion by irradiating the observed portion with normal light, and the observed portion by irradiating the observed portion with special light. Receiving light emitted from the observation unit to obtain a plurality of special images captured at predetermined intervals,
Obtaining the first normal image captured at a first time of the plurality of normal images or the first special image captured at a first time of the plurality of special images; The second normal image captured at a second time point after the first time point or the second special image captured at a second time point after the first time point is acquired. ,
A movement feature amount indicating a change in movement from the observed portion of the first normal image to the observed portion of the second normal image or the second special image of the second special image from the observed portion of the first special image. Acquire the movement feature amount indicating the change of movement to the observed part,
Based on the acquired motion feature amount, a first special image captured at the first time point among the plurality of special images is corrected to obtain a third special image,
An image acquisition method comprising: synthesizing the acquired third special image and the second normal image to acquire a composite image. A light irradiator that irradiates the observed part with normal light and special light; and
The reflected light reflected from the observed part by receiving the normal light is received to pick up a plurality of normal images at a predetermined interval, and the light emitted from the observed part by receiving the special light is received. An imaging unit that captures a plurality of special images at predetermined intervals;
Captured at the first time point of the first normal image or the plurality of special images captured at the first time point of at least one of the plurality of normal images acquired by the imaging unit. In addition to acquiring the first special image, the second normal image captured at the second time point after the first time point or at the second time point after the first time point. The captured second special image is acquired, and a motion feature amount indicating a change in motion from the observed portion of the first normal image to the observed portion of the second normal image or the first special image A motion feature amount acquisition unit that acquires a motion feature amount indicating a change in motion from the observed portion of the image to the observed portion of the second special image;
Based on the motion feature amount acquired by the motion acquisition unit, a motion for correcting the first special image captured at the first time point out of the plurality of special images to acquire a third special image A correction unit;
An image acquisition apparatus comprising: a composite image acquisition unit that acquires a composite image by combining the third special image acquired by the motion correction unit and the second normal image. A display unit for sequentially displaying the plurality of special images captured by the imaging unit in time series;
3. A selection signal receiving unit that receives a signal for selecting at least one special image among the plurality of special images sequentially displayed on the display unit as the first special image. The image acquisition device described. A region-of-interest receiving unit that receives a setting of a region of interest in the predetermined normal image or the predetermined special image;
Pixel signals in each region of interest of the plurality of special images corresponding to the region of interest set in the predetermined normal image or each interest in the plurality of special images corresponding to the region of interest set in the predetermined special image The image acquisition apparatus according to claim 2, further comprising: a past image selection unit that selects the first special image based on a pixel signal in the region. A display unit that simultaneously displays the plurality of special images captured by the imaging unit;
3. A selection signal receiving unit that receives a signal for selecting at least one special image of the plurality of special images simultaneously displayed on the display unit as the first special image. The image acquisition device described. A tube image extraction unit that extracts an image representing a tube portion included in the observed portion from the third special image;
The composite image acquisition unit is configured to acquire a composite image by combining an image representing a tube portion extracted by the tube image extraction unit and the second normal image. The image acquisition device according to any one of 5.   The composite image acquisition unit is configured to acquire a composite image by combining the third special image, the second special image, and the second normal image. The image acquisition device according to claim 1.   The image acquisition apparatus according to claim 1, wherein the first special image is plural.