WO2022082930A1

WO2022082930A1 - Fluorescence microscope system for visible-near-infrared real-time image fusion

Info

Publication number: WO2022082930A1
Application number: PCT/CN2020/132362
Authority: WO
Inventors: 蔡惠明; 王毅庆; 王子阳
Original assignee: 南京诺源医疗器械有限公司
Priority date: 2020-10-22
Filing date: 2020-11-27
Publication date: 2022-04-28
Also published as: CN112014965A

Abstract

A fluorescence microscope system for visible-near-infrared real-time image fusion. The system comprises a built-in laser coaxial illumination module (1), a visible light transmission illumination module, a visible light-near-infrared fluorescence double-channel synchronous imaging module, a data line (13) and an industrial control computer (14). A laser of 785 nm emitted by a solid laser device (101) sequentially passes through an optical fiber bundle (102), a collimating lens (103), a laser beam expander (2), a spectroscope (3) and an objective lens (5) and then reaches a sample (17); visible light emitted by a mercury lamp (15) of the visible light transmission illumination module irradiates the sample (17); visible light emitted by the sample (17) sequentially passes through the spectroscope (3), a 785 nm notch filter (6), a first reflecting mirror (7), an imaging lens (8), a prism (9) and a second reflecting mirror (10) and then reaches an integrated camera (12); near-infrared fluorescence emitted by the sample (17) sequentially passes through the spectroscope (3), the 785 nm notch filter (6), the first reflecting mirror (7), the imaging lens (8), the prism (9) and a 800 nm long wave pass filter (11) and then reaches the integrated camera (12); and the integrated camera (12) is in signal connection with the industrial control computer (14) by means of the data line (13).

Description

A Visible-NIR Real-time Image Fusion Fluorescence Microscope System

technical field

The invention relates to the medical field, and more particularly, to a fluorescence microscope system for visible-near-infrared real-time image fusion.

Background technique

Fluorescence microscopy is to detect the spatial distribution of fluorescent substances in the sample to detect the structure and composition information on the surface and interior of the sample, especially in the field of biomedical fluorescence labeling microscopy detection has been widely used. Currently, fluorescence microscopes on the market are limited to visible-band fluorescence, and microscopes that can be used to observe near-infrared fluorescence are very few and expensive. Compared with visible fluorescence, near-infrared fluorescence can effectively reduce the background noise of fluorescence imaging and improve the tissue penetration depth due to its lower absorption, scattering and autofluorescence properties of biological tissues. The medical field has broad application prospects. In the clinical practice of near-infrared fluorescence imaging technology, there is an urgent need for a means to study the microscopic distribution of near-infrared fluorescent substances (especially indocyanine green and its derivatives). The distribution of fluorescent substances needs to be compared and analyzed with the conventional visible light dyeing images in a real-time synchronization manner, which puts forward higher requirements for near-infrared fluorescence microscopy, that is, visible-near-infrared real-time image fusion.

At present, most of the solutions for near-infrared fluorescence microscopes are systems that combine near-infrared spectrometers with optical microscopes. Fluorescence images are obtained through point-by-point spectral scanning and two-dimensional image reconstruction. This kind of method has slow imaging speed, large data flow, and requires powerful computer analysis and processing capabilities; the image reconstructed from the analog signal is not a real optical signal, which distorts the spatial distribution of the fluorescence signal presented. The requirements of infrared real-time image fusion, facing the urgent clinical and research needs, it is very necessary to develop a fluorescence microscope with visible-near infrared real-time image fusion.

SUMMARY OF THE INVENTION

The present invention aims to provide a fluorescence microscope system for visible-near-infrared real-time image fusion, and aims to solve the problem that the existing near-infrared fluorescence microscope in the prior art cannot realize visible-near-infrared real-time image fusion, so as to meet the needs of clinical rapid pathology of rapid testing screening needs.

To achieve this purpose, the present invention provides a visible-near-infrared real-time image fusion fluorescence microscope system, including a built-in laser coaxial illumination module, a visible light transmission illumination module, a visible-near-infrared fluorescence dual-channel synchronous imaging module, a data line and a Industrial computer; the built-in laser coaxial lighting module includes solid-state laser, optical fiber bundle, collimating lens, laser beam expander, beam splitter and objective lens. The 785nm laser emitted by the solid-state laser passes through the optical fiber bundle, collimating lens, and laser beam expander in sequence. The visible light transmission illumination module includes a mercury lamp, and the visible light emitted by the mercury lamp illuminates the sample; the visible-near-infrared fluorescence dual-channel synchronous imaging module includes a 785nm notch filter, a first reflector, Imaging lens, prism, second reflector, 800nm long-pass filter, integrated camera, data line and industrial computer, of which 785nm notch filter is used to absorb 785nm laser and allow visible light and near-infrared fluorescence to pass through, prism used The 800nm long-pass filter is used to purify the fluorescence; the visible light emitted by the sample passes through the beam splitter, the 785nm notch filter, the first reflector, the imaging lens, the prism and the first in turn. The two mirrors reach the integrated camera; the near-infrared fluorescence emitted by the sample sequentially passes through the beam splitter, the 785nm notch filter, the first mirror, the imaging lens, the prism and the 800nm long-pass filter to reach the integrated camera; the integrated camera passes through the data line Connect with the industrial computer signal.

Preferably, the built-in laser coaxial illumination module further includes a 785nm bandpass filter for filtering the 785nm laser light transmitted from the laser beam expander.

Preferably, the visible light transmission lighting module further includes a 770 nm short-wave-pass filter for filtering the light emitted by the mercury lamp.

Preferably, the laser beam expander includes a concave lens, a convex lens and a sleeve, and both the concave lens and the convex lens are located in the sleeve.

Preferably, the optical fiber bundle includes a plurality of optical fibers with rounded corners and rectangular cross-sections.

Preferably, the integrated camera integrates a black and white camera and a color camera.

Preferably, the beam splitter is a 780 nm notch dichroic mirror.

Compared with the prior art, the advantages of the present invention are: through a system design of a fluorescence microscope using visible-near-infrared real-time image fusion with built-in coaxial illumination, a built-in laser coaxial illumination method is adopted, combined with a 45° setting. The excitation light band notch dichroic mirror and the excitation light band notch filter set at 0° realize the separation of the excitation light from other wavelength bands (including visible light and fluorescence), and then preliminarily separate the visible light and the fluorescence through the prism 9, 770nm The short-pass filter 16 and the 800nm long-pass filter 11 further separate visible light and fluorescence, and then simultaneously image on an integrated camera 12 that integrates a black-and-white camera and a color camera, and finally uses image real-time fusion technology to fuse the fluorescence image with false color. The form of the image is superimposed on the visible light image in real time, and the static quantitative analysis of the image fluorescence intensity and the real-time measurement of the size in the image can be performed. The invention is suitable for real-time dual-channel display of the distribution of near-infrared fluorescent substances (especially suitable for indocyanine green and its derivatives) with an excitation peak of 785±5nm and an emission peak of 800-900nm in biological samples such as biological tissues. Micro-observation can meet the needs of rapid clinical detection and screening.

Description of drawings

FIG. 1 is a schematic diagram of a fluorescence microscope system for visible-near-infrared real-time image fusion of the present invention.

Description of the labels in the figure:

1. Laser coaxial illumination module; 2. Laser beam expander; 3. 785nm bandpass filter; 4. Beam splitter; 5. Objective lens; 6. 785nm notch filter; 7. First reflector; 8 , Imaging lens; 9, Prism; 10, Second mirror; 11, 800nm long-pass filter; 12, Integrated camera; 13, Data cable; 14, Industrial computer; 16, 770nm short-pass filter; 17, Sample; 101, solid-state laser; 102, fiber bundle; 103, collimating lens; 201, concave lens; 202, convex lens.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention; obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. The embodiments of the present invention, and all other embodiments obtained by those of ordinary skill in the art without creative work, fall within the protection scope of the present invention.

In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "upper", "lower", "inner", "outer", "top/bottom", etc. are based on the orientations shown in the accompanying drawings Or the positional relationship is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed to indicate or imply relative importance.

In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "installation", "provided with", "sleeve/connection", "connection", etc., should be understood in a broad sense, such as " Connection", which can be a fixed connection, a detachable connection, or an integral connection, a mechanical connection, an electrical connection, a direct connection, or an indirect connection through an intermediate medium, or an internal connection between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in specific situations.

Please refer to Figure 1, a fluorescence microscope with visible-near-infrared real-time image fusion, including a built-in laser coaxial illumination module 1, a visible light transmission illumination module, a visible-near-infrared fluorescence dual-channel synchronous imaging module, a data line 13 and an industrial control Machine 14; the built-in laser coaxial illumination module includes a solid-state laser 101, an optical fiber bundle 102, a collimating lens 103, a laser beam expander 2, a beam splitter 4 and an objective lens 5, and the 785nm laser emitted by the solid-state laser 101 sequentially passes through the optical fiber bundle 102 , the collimating lens 103, the laser beam expander 2, the beam splitter 4 and the objective lens 5 reach the sample 17; the visible light transmission illumination module includes a mercury lamp 15, and the visible light emitted by the mercury lamp 15 illuminates the sample 17; visible light-near-infrared fluorescence dual-channel synchronization Imaging module, including 785nm notch filter 6, first reflector 7, imaging lens 8, prism 9, second reflector 10, 800nm long-pass filter 11, integrated camera 12, data line 13 and industrial computer 14 , the 785nm notch filter 6 is used to absorb the 785nm laser and allow visible light and near-infrared fluorescence to pass through, the prism 9 is used to totally reflect visible light and enhance the near-infrared fluorescence transmission, and the 800nm long-pass filter 11 is used to purify the fluorescence The visible light emitted by the sample 17 reaches the integrated camera 12 via the beam splitter 4, the 785nm notch filter 6, the first reflector 7, the imaging lens 8, the prism 9 and the second reflector 10 in turn; the near-infrared fluorescence emitted by the sample 17 The integrated camera 12 reaches the integrated camera 12 through the beam splitter 4, the 785nm notch filter 6, the first mirror 7, the imaging lens 8, the prism 9 and the 800nm long-pass filter 11 in turn; the integrated camera 12 communicates with the industrial computer 14 via the data line 13 signal connection.

Preferably, the built-in laser coaxial illumination module 1 further includes a 785nm bandpass filter 3 for filtering the 785nm laser light transmitted from the laser beam expander 2; the visible light transmission illumination module further includes a 770nm short-wavepass filter 16, for filtering the light emitted by the mercury lamp 15.

In addition, the laser beam expander 2 includes a concave lens 201, a convex lens 202 and a sleeve, and both the concave lens 201 and the convex lens 202 are located in the sleeve; the optical fiber bundle 102 includes a plurality of optical fibers with rounded rectangular cross-sections.

The laser light emitted by the solid-state laser 101 reaches the sample 17 through the optical fiber bundle 102, the collimating lens 103, the laser beam expander 2, the beam splitter 4 and the objective lens 5 in sequence, and the light (preferably white light) emitted by the mercury lamp 15 also irradiates the sample 17. On, sample 17 is excited to emit near-infrared fluorescence, and sample 17 also reflects visible light and laser light; the near-infrared fluorescence leaving sample 17 is accompanied by visible light and interfering laser light, enters objective lens 5, and is then reflected by a 780nm notch dichroic mirror Most of the laser light is removed, visible light and fluorescent light and very little laser light are allowed to pass, and then the remaining laser light is further filtered out by the 785nm notch filter 6, and visible light and fluorescent light are allowed to pass.

Further, the integrated camera 12 integrates a black and white camera and a color camera; the beam splitter is a 780nm notch dichroic mirror. The 780nm notch dichroic mirror reflects the 785nm laser light and allows visible light and near-infrared fluorescence to pass, so the fluorescence generated by the sample 17 passes through the objective lens 5, the 780nm notch dichroic mirror, the 785nm notch filter 6, the first reflection in sequence The mirror 7, the imaging lens 8, the prism 9, the 800 nm long-pass filter 11 (fluorescence purification) reach the black and white camera of the integrated camera 12 and form a black and white image.

Similarly, the visible light generated by the sample 17 reaches the color of the integrated camera 12 through the 780nm notch dichroic mirror, the 785nm notch filter 6, the first reflecting mirror 7, the imaging lens 8, the prism 9 and the second reflecting mirror 10 in sequence. camera and form a color image.

Finally, the integrated camera 12 transmits the black and white imaging and color imaging to the industrial computer 14 through the data line 13, and uses the real-time image fusion technology to superimpose the fluorescent image on the visible light image in the form of pseudo-color in real time, and the fluorescence intensity of the image can be calculated in real time. Static quantitative analysis and real-time measurement of dimensions in drawings.

The specific imaging and real-time image fusion method includes the following steps:

Step S1: the white light of the mercury lamp 15 irradiates the sample 17, and the laser light emitted from the solid-state laser 101 is also irradiated on the sample 17 to excite near-infrared fluorescence, while reflecting visible light and laser light;

Step S2: The fluorescence leaving the sample 17 is accompanied by visible light and interfering laser light, enters the objective lens 5, and then passes through the 780nm notch dichroic mirror set at 45° to reflect most of the laser light, allowing visible light and fluorescence and very little laser light. Pass, and then pass through the 785nm notch filter 6 to further filter the remaining laser light, and allow visible light and fluorescence to pass;

Step S3 : the visible light and the fluorescence are reflected by the first reflecting mirror 7 and enter the imaging lens 8 , the visible light and the fluorescence are separated by the prism 9 , wherein the fluorescence is purified by the 800 nm long-wave filter 11 and then reaches the integrated camera 12 , and the visible light directly reaches the integrated camera 12 ;

Step S4: The industrial computer reads the color image and the fluorescent black-and-white image through the data line, and converts the grayscale value x of the black-and-white image into the green component value G in real time, and the red, green, and blue components R', G', and B' of the color image are respectively Multiply by the fusion coefficient α, then add the green component G multiplied by (1-α), and the red, green, and blue components of the final real-time fusion image are αR', αG'+(1-α)G, αB';

Step S5: In order to reduce the background green and enhance the display of weak fluorescence, the fluorescence threshold t and the fluorescence enhancement coefficient e are increased, and can be adjusted by the user within a certain range to obtain the red, green and blue components of the corrected real-time fusion image. is αR', αG'+e(1-α)(G-t), αB';

Step S6: In order to display the fluorescence distribution more intuitively, the grayscale value x of the black and white image is converted into pseudo-color in real time, and the red, green, and blue components of the pseudo-color are the functions R(x), G(x), B(x), the specific function is optimized with reference to the different pseudo-color color schemes of Look-Up-Table in the open source software Image-J;

Step S7: Display the visible light image, fluorescent black and white image, real-time fusion image, and pseudo-color image on the computer screen at the same time, the main window is one of the four images, and the sub-window is all four images, which can be switched by clicking the sub-window screen Main window display screen;

Step S8: In order to realize real-time quantitative analysis of the fluorescence intensity, the method of taking the base point value is used to quantify the fluorescence intensity at different positions. Specifically, after a still image, the average value of the fluorescence CMOS grayscale values of the pixels in the 10x10 to 30x30 square around the target point is taken as base point value;

Step S9: In order to realize the function of real-time measurement of the size in the picture, place a microscope micrometer under the microscope to capture the visible light images of different objective lenses under 5 magnifications in advance, and obtain the average corresponding to each pixel under the 5 magnifications of different objective lenses through Image-J analysis. Size γμm, and then manually select the current objective lens 5 magnification in actual use, and after the still image, count the pixels on the line segment n on the selected line segment, then obtain the line segment size nγμm under the 5 magnification of the corresponding objective lens, or calculate the in-plane pixels of the selected area. Counting m, the area mγ ² μm ² corresponding to the objective lens at a magnification of 5 is obtained.

The present invention provides a system design of a fluorescence microscope using visible-near-infrared real-time image fusion with built-in coaxial illumination, adopts built-in laser coaxial illumination, combined with a 45°-set excitation light band notch dichroic mirror and The excitation light band notch filter set at 0° realizes the separation of excitation light and other wavelength bands (including visible light and fluorescence), and then the visible light and fluorescence are preliminarily separated by prism 9, 770nm short-wave filter 16 and 800nm long-wave filter The filter 11 further separates visible light and fluorescence, and then simultaneously performs imaging on the integrated camera 12 that integrates a black-and-white camera and a color camera, and finally uses the real-time image fusion technology to superimpose the fluorescence image on the visible light image in real time in the form of pseudo-color, It can perform static quantitative analysis of the image fluorescence intensity and real-time measurement of the size in the image. The invention is suitable for real-time dual-channel display of the distribution of near-infrared fluorescent substances (especially suitable for indocyanine green and its derivatives) with an excitation peak of 785±5nm and an emission peak of 800-900nm in biological samples such as biological tissues. Micro-observation can meet the needs of rapid clinical detection and screening.

The above description is only a preferred embodiment of the present invention; however, the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical scope of the present invention, according to the technical solution of the present invention and its improvement concept, equivalently replaces or changes, should be covered within the protection scope of the present invention.

Claims

A fluorescence microscope system for visible-near-infrared real-time image fusion, characterized in that it comprises a built-in laser coaxial illumination module (1), a visible-light transmission illumination module, a visible-near-infrared fluorescence dual-channel synchronous imaging module, and a data line (13) and an industrial computer (14); the built-in laser coaxial illumination module (1) includes a solid-state laser (101), an optical fiber bundle (102), a collimating lens (103), a laser beam expander (2), a beam splitter (4) and The objective lens (5), the 785nm laser light emitted by the solid-state laser (101) reaches the optical fiber bundle (102), the collimating lens (103), the laser beam expander (2), the beam splitter (4) and the objective lens (5) in sequence A sample (17); a visible light transmission illumination module, comprising a mercury lamp (15), and the visible light emitted by the mercury lamp (15) illuminates the sample (17); a visible light-near-infrared fluorescence dual-channel synchronous imaging module, comprising a 785nm notch filter ( 6), a first reflector (7), an imaging lens (8), a prism (9), a second reflector (10), an 800nm long-wave filter (11), an integrated camera (12), and a data cable (13) ) and an industrial computer (14), wherein the 785nm notch filter (6) is used to absorb the 785nm laser and allow visible light and near-infrared fluorescence to pass through, the prism (9) is used to totally reflect visible light and enhance the near-infrared fluorescence transmission, 800nm The long-pass filter (11) is used for purifying the fluorescence; the visible light emitted by the sample (17) passes through the spectroscope (4), the 785nm notch filter (6), the first reflection mirror (7), the imaging lens ( 8), the prism (9) and the second mirror (10) reach the integrated camera (12); the near-infrared fluorescence emitted by the sample (17) passes through the beam splitter (4), the 785nm notch filter (6), the A reflecting mirror (7), imaging lens (8), prism (9) and 800nm long-wave pass filter (11) reach the integrated camera (12); the integrated camera (12) communicates with the industrial computer (14) via the data line (13) ) signal connection.
The fluorescence microscope system for visible-near-infrared real-time image fusion according to claim 1, wherein the built-in laser coaxial illumination module (1) further comprises a 785nm bandpass filter (3) for The 785nm laser light from the laser beam expander (2) is filtered.
The fluorescence microscope system for visible-near-infrared real-time image fusion according to claim 1, characterized in that: the visible light transmission illumination module further comprises a 770 nm short-wave filter (16) for filtering the mercury lamp (15) emitted of light.
The visible-near-infrared real-time image fusion fluorescence microscope system according to claim 1, characterized in that: the laser beam expander (2) comprises a concave lens (201), a convex lens (201) and a sleeve, and the concave lens (201) ) and the convex lens (201) are located in the sleeve.
The fluorescence microscope system for visible-near-infrared real-time image fusion according to claim 1, characterized in that: the optical fiber bundle (102) comprises a plurality of optical fibers with rounded rectangular cross-sections.
The visible-near-infrared real-time image fusion fluorescence microscope system according to claim 1, wherein the integrated camera (12) integrates a black-and-white camera and a color camera.
The fluorescence microscope system for visible-near-infrared real-time image fusion according to claim 1, wherein the beam splitter (4) is a 780 nm notch dichroic mirror.