WO2016035450A1

WO2016035450A1 - Imaging device

Info

Publication number: WO2016035450A1
Application number: PCT/JP2015/069596
Authority: WO
Inventors: 足立　晋; 木村　健士
Original assignee: 株式会社島津製作所
Priority date: 2014-09-01
Filing date: 2015-07-08
Publication date: 2016-03-10

Abstract

A fluorescent image processing unit (32) corrects and stores a dark image which is output from a fluorescence imaging element (52) without fluorescence impinging on the fluorescence imaging element (52), and subtracts the corrected dark image from a fluorescent image captured by the fluorescence imaging element (52).

Description

Imaging device

The present invention relates to an imaging apparatus for irradiating a fluorescent dye injected into a subject with excitation light and photographing fluorescence generated from the fluorescent dye.

In recent years, a technique called near-infrared fluorescence imaging has been used for surgery. In this near-infrared fluorescence imaging, indocyanine green (ICG) as a fluorescent dye is injected into the affected area. When indocyanine green is irradiated with near infrared light of 750 to 810 nm (nanometer) as excitation light, indocyanine green emits fluorescence in the near infrared region having a peak at about 850 nm. This fluorescence is photographed by a camera capable of detecting near infrared light, and the image is displayed on a display unit such as a liquid crystal display panel. According to this near-infrared fluorescence imaging, it is possible to observe blood vessels, lymph vessels, and the like existing at a depth of about 20 mm from the body surface.

Patent Document 1 discloses a near-infrared fluorescence intensity distribution image obtained by irradiating an indocyanine green excitation light to a living organ to which indocyanine green is administered, and indocyanine green administration. Compared with the cancer lesion distribution image obtained by applying X-rays, nuclear magnetic resonance or ultrasound to the previous test organ, it is detected by the intensity distribution image of near-infrared fluorescence, A data collection method is disclosed in which data of a region that is not detected in a cancer lesion distribution image is collected as cancer secondary lesion region data.

Patent Document 2 discloses that a subject to which an angiographic contrast agent is administered is alternately irradiated with excitation light and visible light, and a fluorescent image and a visible image irradiated with excitation light by an imaging unit are alternately displayed. A surgical support method is disclosed in which a blood vessel image is extracted by performing threshold processing on a fluorescent image with a predetermined threshold, and a composite image in which a visible image and the extracted blood vessel image are superimposed is obtained.

International Publication No. 2009/139466 JP 2009-226072 A

Fluorescence from indocyanine green is very weak compared to visible light. For this reason, when photographing this fluorescence with a CMOS or CCD camera, it is necessary to take measures to increase the excitation light emitted from the excitation light source or to increase the amplification factor of the camera that photographs the excitation light. There is a need.

However, when the excitation light is strengthened, there is a problem that not only the power consumption increases, but also the amount of heat generated during photographing increases. On the other hand, when the amplification factor of the camera is increased, a fixed pattern peculiar to the camera is amplified, resulting in a problem that an artifact is generated.

For example, when a CMOS sensor is used as a camera, artifacts caused by a readout circuit are generally superimposed on the sensor output. This is caused by variations in characteristics of FETs (field effect transistors), conversion errors of built-in ADCs (analog / digital conversion circuits), and the like. Such artifacts are particularly problematic when photographing fluorescence that is weak compared to visible light.

The present invention has been made to solve the above problems, and even when photographing fluorescence, it is possible to prevent occurrence of artifacts due to the image sensor and display an accurate fluorescence image. An object is to provide an imaging apparatus.

In the first aspect of the invention, an excitation light source for irradiating the subject with an excitation light beam for exciting the fluorescent dye injected into the subject, and fluorescence generated from the fluorescent dye by irradiating the excitation light beam An imaging device capable of capturing an image, and correcting a dark image that is an output of the imaging device in a state where the fluorescence is not incident on the imaging device in an imaging apparatus that captures a fluorescent image of the subject at a predetermined frame rate An image correction unit, a dark image storage unit that stores a dark image corrected by the image correction unit, and a dark image stored in the dark image storage unit from a fluorescence image captured by the imaging element; And an image processing unit that transmits the subtracted image as a display image to the display unit.

In the second invention, the image correction unit adds a preset offset value to the pixel value of the dark image.

In the third invention, the image correction unit adds the offset value to the pixel value of the image sensor that has captured the fluorescent image of the subject.

In the fourth invention, the image correction unit corrects the dark image so that the pixel value of the dark image after correction is in the vicinity of the lower end of the dynamic range of the output value of the image sensor.

In the fifth invention, the dark image is acquired at a constant interval with respect to an image acquired from the image sensor at a predetermined frame rate, and the dark image acquired at a constant interval is integrated and averaged. It is stored in the dark image storage unit.

In a sixth aspect of the invention, the image processing unit includes a display image storage unit that stores the display image, and the image processing unit replaces the dark image with the display when the dark image is acquired from the imaging element. The display image stored in the image storage unit is transmitted to the display unit.

In a seventh aspect of the invention, the image processing device further includes a second imaging element capable of capturing a visible light image of the subject, and the image processing unit receives the display image and the visible image captured by the second imaging element. Send to display.

According to the first to fourth aspects of the invention, even when fluorescence is photographed, it is possible to prevent occurrence of artifacts due to the image sensor and display an accurate fluorescence image.

According to the fifth aspect of the invention, even when the dark image changes due to a temperature change or the like, it is possible to prevent the occurrence of artifacts and display an accurate fluorescent image.

According to the sixth aspect of the invention, it is possible to prevent a dark image from being mixed in an image displayed at a predetermined frame rate on the display unit, and to make a display image suitable.

According to the seventh aspect, a visible image and an accurate fluorescent image can be displayed on the display unit.

1 is a schematic diagram of an imaging apparatus according to the present invention. 2 is a perspective view of an illumination / photographing unit 12. FIG. 2 is a schematic diagram of a camera 21. FIG. It is a block diagram which shows the main control systems of the imaging device which concerns on this invention. It is explanatory drawing which shows the fundamental view of this invention. It is explanatory drawing which shows the correction method of the dark image D typically. 3 is a functional block diagram of a fluorescence image processing unit 32. FIG. It is a circuit diagram of an addition circuit.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a schematic diagram of an imaging apparatus according to the present invention.

The imaging apparatus includes an input unit 11 such as a touch panel, and includes a main body 10 incorporating a control unit 30 and the like described later, an illumination / photographing unit 12 supported movably by an arm 13, and a liquid crystal display panel. A display unit 14 and a treatment table 16 on which a patient 17 is placed are provided. The illumination / imaging unit 12 is not limited to the one supported by the arm 13, and may be carried by the surgeon in hand. The display unit 14 may be a glasses-type wearable device instead of a liquid crystal display panel.

FIG. 2 is a perspective view of the illumination / photographing unit 12 described above.

The illumination / photographing unit 12 includes a camera 21 capable of detecting near-infrared rays and visible light, a visible light source 22 including a large number of LEDs disposed on the outer peripheral portion of the camera 21, and an outer peripheral portion of the visible light source 22. And an excitation light source 23 made up of a number of arranged LEDs. The visible light source 22 emits visible light. The excitation light source 23 irradiates near infrared light having a wavelength of 760 nm, which is excitation light for exciting indocyanine green.

FIG. 3 is a schematic diagram of the camera 21.

The camera 21 includes a movable lens 71 that reciprocates for focusing, a wavelength selection filter 53, a visible light image sensor 51, and a fluorescence image sensor 52. The visible light image sensor 51 and the fluorescence image sensor 52 are composed of a CMOS or a CCD. Visible light and fluorescence incident on the camera 21 coaxially along the optical axis L pass through the movable lens 71 constituting the focusing mechanism and then reach the wavelength selection filter 53. Of visible light and fluorescence incident coaxially, visible light is reflected by the wavelength selection filter 53 and enters the visible light image sensor 51. Of the coaxial visible light and fluorescence, the fluorescence passes through the wavelength selection filter 53 and enters the fluorescence imaging device 52. At this time, by the action of the focusing mechanism including the movable lens 71, the visible light is focused on the visible light image sensor 51 and the fluorescence is focused on the fluorescence image sensor 52.

FIG. 4 is a block diagram showing a main control system of the imaging apparatus according to the present invention.

This imaging apparatus is composed of a CPU that performs logical operations, a ROM that stores operation programs necessary for controlling the apparatus, a RAM that temporarily stores data during control, and the like, and a control unit that controls the entire apparatus 30. The control unit 30 synthesizes the visible image and the fluorescent image with the visible image processing unit 31 for processing the visible image, the fluorescent image processing unit 32 for processing the fluorescent image generated from indocyanine green, and the like. Image synthesizing unit 33.

The control unit 30 is connected to the input unit 11 and the display unit 14 described above. The control unit 30 is connected to an illumination / photographing unit 12 including a camera 21, a visible light source 22, and an excitation light source 23. The visible image processing unit 31 in the control unit 30 transmits a control signal to the visible light source 22 and the visible light imaging device 51 and receives an image signal of a visible image from the visible light imaging device 51. Further, the fluorescence image processing unit 32 in the control unit 30 transmits a control signal to the excitation light source 23 and the fluorescence imaging element 52 and receives an image signal of the fluorescence image from the fluorescence imaging element 52.

When performing an operation using the imaging apparatus having the above-described configuration, indocyanine green is injected into the patient 17 who is supine on the treatment table 16 by injection. Then, near infrared rays are emitted from the excitation light source 23 toward the subject including the affected part, and visible light is emitted from the visible light source 22. As the near infrared light emitted from the excitation light source 23, as described above, 760 nm near infrared light acting as excitation light for indocyanine green to emit fluorescence is employed. As a result, indocyanine green generates fluorescence in the near infrared region having a peak at about 850 nm.

Then, the camera 21 captures the vicinity of the affected part of the patient 17. As described above, the camera 21 can detect fluorescence and visible light. An image obtained by irradiating the patient 17 with visible light and photographing this with the camera 21 becomes a visible image, and an image obtained by irradiating the patient 17 with near-infrared light and photographing fluorescence from indocyanine green with the camera 21 is a fluorescent image. Become. A fluorescent image captured by the camera 21 at a predetermined frame rate is processed by the fluorescent image processing unit 32, and a visible image captured by the camera 21 at a predetermined frame rate is processed by the visible image processing unit 31. Then, the processed fluorescence image data and visible image data are synthesized by the image synthesis unit 33 to create a fusion image in which the visible image and the fluorescence image are fused. On the display unit 14, a fluorescent image, a visible image, and a fusion image are displayed simultaneously or selectively in divided areas.

In such a configuration, artifacts caused by the readout circuit are generally superimposed on the images photographed by the visible light image sensor 51 and the fluorescence image sensor 52. The presence of this artifact becomes a problem when the signal amplification factor of the fluorescence imaging element 52 is increased, particularly when photographing fluorescence from indocyanine green which is much weaker than visible light.

For this reason, in the imaging apparatus according to the present invention, the fluorescence image processing unit 32 corrects and stores the dark image that is the output of the fluorescence imaging element 52 in a state where the fluorescence is not incident on the fluorescence imaging element 52, and stores the fluorescence image. By subtracting the corrected dark image from the fluorescence image picked up by the image pickup device 52, the above-described artifact problem is solved.

FIG. 5 is an explanatory diagram showing the basic concept of the present invention.

The fluorescence imaging device 52 receives the control signal from the fluorescence image processing unit 32 and acquires the fluorescence image P at a predetermined frame rate. Then, for each predetermined frame, instead of the fluorescence image P, a dark image D that is an output of the fluorescence image sensor 52 in a state where no fluorescence enters the fluorescence image sensor 52 is acquired. In the embodiment shown in FIG. 5, a case where one of 10 frames is a dark image D is shown. After the dark image D is corrected, the corrected dark image D is subtracted from the fluorescent image P. Note that the fluorescent image P and the dark image D are used after recursive computation, as will be described later.

FIG. 6 is an explanatory diagram schematically showing a method of correcting the dark image D.

In this figure, the horizontal axis indicates a predetermined area of the fluorescence imaging element 52, and the vertical axis indicates the pixel value of the dark image D in that area. In general, since a dark image is designed so that its dark level is zero, a region where a negative pixel value is originally fixed is zero. For this reason, the pixel values in the originally negative region hatched in FIG. 6A are not considered. For this reason, even if the dark image D is simply subtracted from the fluorescence image P captured by the fluorescence imaging element 52, the artifact cannot be completely eliminated.

For this reason, in the imaging apparatus according to the present invention, a preset offset value is added to the pixel value of the dark image D using an adder circuit described later, as shown in FIG. A correction configuration is adopted. At this time, in order to prevent an adverse effect due to the excessive offset value, the extent to which the negative region in the pixel value of the dark image D is eliminated, that is, the pixel value of the corrected dark image D is equal to the image sensor for fluorescence. The configuration in which the dark image D is corrected so as to be near the lower end of the dynamic range of the output value of 52 is employed. This offset value may be set when the apparatus is activated, or may be changed over time at regular intervals.

Note that the dark image D in this specification is an output of the fluorescence imaging element 52 in a state where no fluorescence is incident on the fluorescence imaging element 52. The dark image D may be an output of the fluorescence imaging element 52 when the excitation light source 23 is turned off, or may be an output of the fluorescence imaging element 52 when the fluorescence is blocked by a shutter or the like. Furthermore, a state in which fluorescence is not incident electronically may be created by setting the exposure time (charge accumulation time) of the fluorescence imaging element 52 to zero. This dark image D may be one frame of data, or may be a simple addition average or a weighted addition average of a plurality of frames of data.

FIG. 7 is a functional block diagram of the fluorescent image processing unit 32 described above.

The fluorescent image processing unit 32 includes an image memory 44 for storing the dark image D, an image memory 45 for storing the fluorescent image after dark image subtraction,

computing units

41, 42, and 43, and a bus controller 46. With. The

computing units

41, 42, 43 and the bus controller 46 are connected by a data bus 47.

FIG. 8 shows a circuit of an adding circuit as an image correcting unit according to the present invention that generates a corrected dark image D and fluorescent image P by adding a preset offset value to the output from the fluorescence imaging device 52. FIG.

This adding circuit adds an offset value to the dark image D by adding the offset voltage Vr to the output voltage Vsig from the fluorescent imaging device 52, as shown in FIG. This is for adding a similar offset value to the fluorescent image P. The addition circuit includes an addition amplifier 65, an A / D converter 66, a D / A converter 67, and a switch 61 that closes to the Vsig side when receiving the output voltage Vsig from the fluorescence imaging element 52. A switch 68 that is closed from the D / A converter 67 to the offset voltage output (Vr) side, a reset switch 62, and

capacitors

63, 64, and 69 are provided. In this adder circuit, the offset voltage Vr is added from the D / A converter 67 to the output voltage Vsig from the fluorescence imaging element 52, and then the added voltage is converted into a pixel value by the A / D converter 66. It has a configuration for outputting as data. Note that only one adder circuit may be provided for the fluorescence image sensor 52, and a plurality of adder circuits are built in and integrated for each signal readout row (or readout column) inside the fluorescence image sensor 52. It may be arranged.

In the embodiment described above, the offset value is added to both the dark image D and the fluorescence image P output from the fluorescence image sensor 52. Thereby, the fluorescence image P can be displayed on the display unit 14 with the same brightness. However, when the luminance of the fluorescent image displayed on the display unit 14 may be reduced to some extent, the addition of the offset value to the fluorescent image may be omitted.

When an image is displayed by the imaging apparatus having the above configuration, the corrected dark image is stored in the image memory 44 shown in FIG. The corrected dark image is a dark image after the offset value is added by the adding circuit shown in FIG. The image memory 44 stores a dark image Dn that is integrated and averaged from the newly acquired dark image D and the previously acquired dark image [Dn−1] according to the following equation (1). In the following formula (1), α is a coefficient that is greater than 0 and equal to or less than 1. This calculation is executed by the calculator 41.
Dn = αD + (1−α) · [Dn−1] (1)

Then, the computing unit 43 subtracts the dark image Dn stored in the image memory 44 from the fluorescent image P acquired by the fluorescent imaging element 52 as shown in the following equation (2), thereby displaying the display image Q. Create The display image Q is sent to the display unit 14 via the image composition unit 33. As a result, it is possible to display an accurate fluorescent image in which artifacts due to the readout circuit of the fluorescent imaging element 52 are prevented.
Q = P−Dn (2)

In parallel with this, the image memory 45 stores the newly acquired display image Q and the display image [Qn−1] acquired in the past after the recursive calculation according to the following equation (3). A display image Qn is stored. In the following formula (3), β is a coefficient that is greater than 0 and equal to or less than 1. This calculation is executed by the calculator 42.
Qn = βQ + (1-β) · [Qn-1] (3)

Then, when the dark image D is acquired from the fluorescence image sensor 52, the fluorescent image processing unit 32 displays the recursive display image Qn stored in the image memory 45 instead of the dark image D on the display unit 14. Send. That is, as shown in FIG. 5, the fluorescence imaging element 52 acquires the fluorescence image P at a predetermined frame rate, and acquires the dark image D instead of the fluorescence image P every predetermined frame. In this case, when the dark image D is acquired from the fluorescence imaging device 52, the display image Qn after recursive stored in the image memory 45 instead of the dark image D is transmitted to the display unit 14, It is possible to prevent the dark image D from being mixed in the image displayed at the predetermined frame rate on the display unit 14 and to make the display image suitable.

In addition, although it is possible to reduce random noise by performing the recursive calculation according to the above-described equation (3), this recursive calculation may be omitted. Here, when performing a recursive calculation, it is preferable to make the above-mentioned count β larger than α. For example, by setting α to about 0.02 and β to about 0.5, it is possible to obtain a suitable image with little afterimage.

In the embodiment described above, a light source that emits near-infrared light having a wavelength of 760 nm is used as the excitation light source 23. However, as the excitation light source 23, excitation light is obtained by exciting indocyanine green. Those that irradiate near-infrared light having a wavelength of about 750 nm to 810 nm capable of generating light can be used.

Further, in the above-described embodiment, indocyanine green is used as a material containing a fluorescent dye, and the indocyanine green is irradiated with near-infrared light of 760 nm as excitation light, so that it is approximately 850 nm from indocyanine green. Although the case of emitting fluorescence in the near-infrared region having a peak at, light other than near-infrared light may be used.

For example, 5-ALA (5-aminolevulinic acid / 5-aminolevulinic acid) can be used as a fluorescent dye. When 5-ALA is used, 5-ALA that has entered the body of the patient 17 changes to a protoporphyrin IX / PpIX that is a fluorescent substance. When visible light of about 400 nm is irradiated toward the protoporphyrin, red visible light is irradiated as fluorescence from the protoporphyrin. Therefore, when 5-ALA is used, an excitation light source that emits visible light having a wavelength of about 400 nm may be used, and a light source for confirmation emits fluorescence from protoporphyrin. What irradiates red visible light to be used may be used.

DESCRIPTION OF SYMBOLS 10 Main body 11 Input part 12 Illumination and imaging | photography part 13 Arm 14 Display part 16 Treatment table 17 Patient 21 Camera 22 Visible light source 23 Excitation light source 30 Control part 31 Visible image process part 32 Fluorescence image process part 33 Image composition part 44 Image memory 45 Image memory 51 Image sensor for visible light 52 Image sensor for fluorescence 65 Addition amplifier

Claims

An excitation light source that irradiates the subject with an excitation beam for exciting the fluorescent dye injected into the subject;
An imaging device capable of imaging fluorescence generated from the fluorescent dye when irradiated with excitation light, and
In an imaging apparatus that captures a fluorescent image of the subject at a predetermined frame rate,
An image correction unit that corrects a dark image that is an output of the image sensor in a state where the fluorescence does not enter the image sensor;
A dark image storage unit that stores a dark image corrected by the image correction unit;
An image processing unit that subtracts the dark image stored in the dark image storage unit from the fluorescence image captured by the image sensor and transmits the image after subtraction to the display unit as a display image;
An imaging apparatus comprising:
The imaging apparatus according to claim 1, wherein
The image correction unit is an imaging apparatus that adds a preset offset value to a pixel value of a dark image.
The imaging apparatus according to claim 2, wherein
The imaging apparatus, wherein the image correction unit adds the offset value to a pixel value of the imaging element that has captured a fluorescent image of the subject.
The imaging device according to claim 3.
The image correction unit corrects the dark image so that the pixel value of the dark image after correction is in the vicinity of the lower end of the dynamic range of the output value of the image sensor.
The imaging device according to claim 4.
The dark image is acquired at a fixed interval with respect to an image acquired at a predetermined frame rate from the imaging device, and the dark images acquired at a fixed interval are averaged and stored in the dark image storage unit. Imaging device.
The imaging device according to claim 5, wherein
A display image storage unit for storing the display image;
When the dark image is acquired from the imaging device, the image processing unit transmits a display image stored in the display image storage unit instead of the dark image to the display unit.
The imaging apparatus according to any one of claims 1 to 6,
A second imaging device capable of capturing a visible light image of the subject;
The image processing unit is an imaging apparatus that transmits the display image and a visible image captured by the second image sensor to the display unit.