EP3152549A2 - Procédé et moyen d'imagerie multispectrale - Google Patents

Procédé et moyen d'imagerie multispectrale

Info

Publication number
EP3152549A2
EP3152549A2 EP15730081.5A EP15730081A EP3152549A2 EP 3152549 A2 EP3152549 A2 EP 3152549A2 EP 15730081 A EP15730081 A EP 15730081A EP 3152549 A2 EP3152549 A2 EP 3152549A2
Authority
EP
European Patent Office
Prior art keywords
light
image
recorded
intensity
spectral
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP15730081.5A
Other languages
German (de)
English (en)
Other versions
EP3152549B1 (fr
EP3152549C0 (fr
Inventor
Nikolas Dimitriadis
Nikolaos DELIOLANIS
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Universitaet Heidelberg
Original Assignee
Universitaet Heidelberg
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Universitaet Heidelberg filed Critical Universitaet Heidelberg
Publication of EP3152549A2 publication Critical patent/EP3152549A2/fr
Application granted granted Critical
Publication of EP3152549B1 publication Critical patent/EP3152549B1/fr
Publication of EP3152549C0 publication Critical patent/EP3152549C0/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/645—Specially adapted constructive features of fluorimeters
    • G01N21/6456—Spatial resolved fluorescence measurements; Imaging
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/043—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances for fluorescence imaging
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
    • A61B1/0638—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements providing two or more wavelengths
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/14—Arrangements specially adapted for eye photography
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0071—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence by measuring fluorescence emission
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/145—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue
    • A61B5/1455—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue using optical sensors, e.g. spectral photometrical oximeters
    • A61B5/14551—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue using optical sensors, e.g. spectral photometrical oximeters for measuring blood gases
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/02—Details
    • G01J3/10—Arrangements of light sources specially adapted for spectrometry or colorimetry
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/28—Investigating the spectrum
    • G01J3/30—Measuring the intensity of spectral lines directly on the spectrum itself
    • G01J3/32—Investigating bands of a spectrum in sequence by a single detector
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/28—Investigating the spectrum
    • G01J3/44—Raman spectrometry; Scattering spectrometry ; Fluorescence spectrometry
    • G01J3/4406—Fluorescence spectrometry
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/6428—Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes"
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/75—Systems in which material is subjected to a chemical reaction, the progress or the result of the reaction being investigated
    • G01N21/77—Systems in which material is subjected to a chemical reaction, the progress or the result of the reaction being investigated by observing the effect on a chemical indicator
    • G01N21/78—Systems in which material is subjected to a chemical reaction, the progress or the result of the reaction being investigated by observing the effect on a chemical indicator producing a change of colour
    • G01N21/80—Indicating pH value
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/02—Details
    • G01J3/10—Arrangements of light sources specially adapted for spectrometry or colorimetry
    • G01J2003/102—Plural sources
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/02—Details
    • G01J3/10—Arrangements of light sources specially adapted for spectrometry or colorimetry
    • G01J2003/102—Plural sources
    • G01J2003/106—Plural sources the two sources being alternating or selectable, e.g. in two ranges or line:continuum
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N2021/6417—Spectrofluorimetric devices
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/6428—Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes"
    • G01N2021/6439—Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes" with indicators, stains, dyes, tags, labels, marks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/91—Investigating the presence of flaws or contamination using penetration of dyes, e.g. fluorescent ink

Definitions

  • the present invention relates to the multispectral imaging of samples, in particular of biological tissues.
  • the illumination light may be absorbed or scattered. If the tissue contains fluorescent molecules, then the absorbed energy is temporarily stored by setting the molecules at an excited state and then it is released as a photon of longer wavelength.
  • the light intensity from fluorescence is usually many orders of magnitude weaker than the intensity of the reflected excitation light, and it is necessary to separate or block the reflected excitation from the emitted light.
  • Some multispectral imaging systems have a single image sensor and implement a fast switching mechanism between reflectance and fluorescence imaging mode. This can be achieved with use of bandpass excitation and emission filter sets that are mounted on filter wheels or filter cubes that are exchanged fast in order to record reflectance and fluorescence images alternatingly with high frequency. This approach is straightforward and allows the highest throughput of light, but requires mechanically moving parts like filter wheels. Further, depending on the filter configuration, it allows the recording of the intensity of only one fluorophore at a time. Switching filters at near video rate frequencies is technically complex and requires accurate mechanical synchronization with the frame grabbing sequence of the camera.
  • spectrally tunable filters for example liquid crystal tunable filters.
  • the switching between spectral settings suitable for different fluorophores can be very fast ( ⁇ 1 ms), however the transmission throughput of the tunable filters is limited. Furthermore, they are highly sensitive to light transmission angles and light polarization, and are associated with rather high costs.
  • An alternative approach for multispectral imaging is to use multiple sensors, where in front of each sensor a corresponding emission filter is arranged.
  • the light can reach each sensor either by passing through a single objective lens and using an optical beam-splitter arrangement to deliver the light to each sensor, or each sensor can have a separate objective lens.
  • each sensor is paired with a filter that can block the excitation wavelengths and record the emission from one fluorophore [Lucia M.A. Crane et al., et al. J Vis Exp. 2010; (44): 2225.].
  • An additional sensor can record the reflection image with a different imaging path. This concept is simple, but the use of multiple sensors, beam splitters or objective lenses increases the size, the complexity of design and the cost.
  • Another solution for multispectral imaging uses switching between different excitation lights.
  • the object is alternatively illuminated with excitation beams that have a specific excitation spectrum that is blocked by filters to enter into one or more cameras.
  • US 20130286176 Al a single color sensor, a laser excitation to excite fluorescence, and a broadband illumination source that switches on and off, is used. When only the laser excitation source is on, then the sensor can capture the emitted fluorescence, and when the broadband illumination is on, then the sensor can capture the reflected image.
  • This system produces a reflectance image and an image of a fluorochrome, but an observer might visually experience a disturbing flickering due to the on-off switching of the different sources.
  • Yet another approach uses filters with multiple-band pass regions paired with a monochrome sensor.
  • a filter in front of a monochrome sensor blocks the excitation wavelengths to enter into the monochrome sensor.
  • the different fluorophores can be imaged individually with excitation scanning.
  • the filtered multi-component fluorescent light can be split into wavelength dependent paths which are then imaged onto different spatial regions of a monochrome sensor. With this approach it is possible to record multiple channels simultaneously with a monochrome sensor.
  • a color sensors can be used to record the multi-component fluorescent light with a multi-channel (and thus color) sensor.
  • the multi-channel sensor output can then be processed in order to obtain the individual fluorescent components.
  • An additional sensor can be used to record the reflectance image by splitting the reflected excitation light into a different optical path imaging that light on that sensor. This offers multiple fluorescence imaging bands together with the reflectance, but an observer will visually perceive false color representation. Depending on the specific excitation wavelengths, the false perception might not be possible to be corrected even digitally.
  • each channel has a narrow bandpass filter in front of the sensor and the intensity in each individual narrow filter band is computed [US 20120085932 Al].
  • the used filter sets are known as “Pinkel”, “Sedat”, or “Full-multiband” depending on the exact combination of excitation and emission filters used in the specific application.
  • the present invention is made to provide a method and means for multispectral imaging, which avoid the above mentioned problems of the prior art and are simple, quick and costeffective.
  • FIG. 1 describes the general concept of the invention.
  • the inventive system comprises an illumination system 100 that produces and guides light that incorporates spectral and time multiplexing properties to illuminate an object 400.
  • the light emanating from the object 400 is collected and detected or imaged by an imaging system 200 that is comprised of elements like lenses, filters, beam splitters, and light sensor/detector arrays (i.e. cameras), etc.
  • Both the illumination system 100 and the detection system 200 are connected to a controlling and processing unit 300 that controls the operation of the illumination system, synchronizes the operation and grabs the images from the detection system 200, and processes the image data, for further evaluation, display and storage.
  • a display/visualization system (500) displays the decomposed images either separately or simultaneously/in overlay.
  • the illumination system 100 operates in two (or more) alternating phases as shown in fig. 2.
  • the system illuminates the object with light exhibiting a spectral shape with areas of high and low intensities similar to that depicted in fig.2A.
  • fig. 2A and fig. 2B for each illumination phase, the spectrum of illumination and the transmission spectra in front of the two sensor arrays are provided.
  • Various spectral shapes are possible, but it is essential that the light has spectral regions with very low intensity at wavelengths longer than the high intensity regions. In those regions fluorescent light emitted by the object upon excitation with the first light can be detected by the imaging system without detecting relevant amounts of reflected light.
  • phase 2 of illumination the object is illuminated in general with a light that has a different spectral shape, and is preferably exhibiting an approximately complementary structure as shown in fig. 2B.
  • the imaging system 200 comprises of two (or more) imaging channels/paths split by a beam splitter 202 or an equivalent method. Each imaging channel has image sensor arrays 211, 221 to detect and record fluorescence and reflectance images at the different phases.
  • the light reaching the image sensors is spectrally attenuated so that in general the illumination light of phase 1 is attenuated before reaching the imaging sensor 2 and the illumination light of phase 2 is attenuated before reaching the imaging sensor 1.
  • illumination phase 1 the spectral bands of the light reflected from the object are transmitted and detected into detector sensor 221 forming a reflectance image, whereas the fluorescence emission from the object is transmitted and detected into sensor 211 forming a fluorescence image.
  • illumination phase 2 the spectral bands of the light reflected from the object are transmitted and detected in detector sensor 211 forming a reflectance image, whereas the fluorescence emission from the object is transmitted to and detected in sensor 221 forming a fluorescence image.
  • the amount of attenuation before the light reaches each sensor can be approximately estimated such that when a sensor is preferably used to detect fluorescence (i.e. sensor 221 in phase 2, and sensor 211 in phase one) the detected fluorescence signal should preferably be 100 times more than the bleed through of the excitation light.
  • One of the preferred embodiments is shown in fig. 3. It uses two broadband white light sources 111 and 121, which can be switched on and off alternatively, filtered by two complementary multiple bandpass filters 112 and 122. The two beams from these two light sources 111 and 121 are combined with a multiple bandpass polychroic mirror 101. In between those elements, collimation lenses may optionally be placed in order to guide more light to area to be imaged.
  • the imaging system 200 consists of lenses, filters, mirrors, and imaging sensors.
  • the light coming from the object 400 is collected in the detection system 200 with an objective lens 201 (or a system of lenses acting as an objective lens), which for use in open surgeries preferably has a focusing distance of 200 mm.
  • the light after the objective lens is splitted by a multichroic bandpass mirror 202 and splits the light into two paths that these light paths have complementary spectral content.
  • Two complementary multiple bandpass filters 213 and 223 attenuate the beams that are imaged by a multi-channel or multi-color sensor arrays 211 and 221.
  • the processing/controlling unit is consisted of a frame grabber 301, a controlling unit 302, and a processing unit 303, to generate images 501, ... 507.
  • Optional lenses 212, 214, 222, 224 can be used in between the various elements to relay the image to the sensor.
  • the mirror beam splitter (202) and the multi-bandpass filters 213 and 223 are preferably placed in an optically infinity corrected imaging space.
  • the mirror 202 is usually l-3mm thick. In order to maintain the optical imaging quality, the flatness of the dichroic mirror 202 should be ⁇ /10.
  • Interference multiple bandpass filters and polychroic filters are usually manufactured as excitation/emission/mirror filter sets for use in fluorescence microscopy as Sedat, Pinkel or full multiband sets.
  • An example of a four-band filter set which is originally configured for imaging four fluorochromes is shown in fig. 4. This set can be configured to be used both in the illumination and imaging system.
  • the excitation filter can be used in positions 122 and 223, the emission filter in positions 112 and 213, and the polychroic mirror in (101) and (202).
  • Various different combinations of filters, filter sets may be optimized for various fluorochrome applications.
  • Using such a filter set for the two illumination modes means that in phase 1 the excitation filter of the set is used to filter white light from source 121 and in phase 2 the emission filter is used to filter white light from source 111.
  • the polychroic miror 101 is used to combine the beams in one.
  • nominal concentrations of fluorochromes in tissue usually between lOOxlO "9 M to lxlO "3 M) the usual attenuation ratio in the rejection bands of interference multiple bandpass filters of optical density (O.D.) of 6 orders of magnitude is sufficient, however it is expected that in many cases attenuation of 2 or 3 O.D. can be adequate.
  • phase 1 the object 400 is illuminated with spectrally shaped light that is partially reflected, transmitted, and absorbed to excite fluorescence.
  • the reflected excitation light in phase 1 is attenuated by the emission filter 213 in front of sensor 211, which records only the fluorescence emission.
  • the filter 223 in front of sensor 221 transmits the reflected excitation light, so the 221 sensor detects the reflected excitation light in phase 1.
  • phase 2 the object 400 is illuminated with light, which has an approximately complementary spectral shape to the illumination light in phase 1.
  • the illumination light in phase 2 is partially reflected, transmitted and absorbed by the object to also excite
  • phase 1 sensor 211 is recording fluorescence and sensor 221 is recording reflectance images of the object 400.
  • phase 2 sensor 221 is recording fluorescence and sensor 211 is recording fluorescence images of the object 400.
  • a series of sequentially acquired reflectance and fluorescence images is shown in fig. 5B and C.
  • the system can image both fluorescence and reflectance images of (almost) the entire visible spectrum. This means that in every two illumination cycles (phases) the reflectance and the fluorescence images are complementary recorded without missing any relevant or substantial part of all spectral regions. If the frequency of illumination phases and recording of the images is fast enough to accommodate any possible object movement, by combining the reflectance image of each frame with the reflectance image of the previous frame from the opposite sensor, which has a complementary reflectance spectrum, a composed reflectance image of the whole spectrum is created. That is combining the Reflectance 1 with Reflectance 2, Reflectance 2 with Reflectance 3, and so on.
  • This combination method not only increases the spectral coverage of both fluorescence and reflectance but also multiplies the number of spectral measurements per camera.
  • this technique offers surplus spectral imaging (for comparably small changes). Only few small spectral regions will not be recorded due to a practical implementation problem that a small spectral gap may be necessary to exist between complementary spectral areas. Though, those spectral gaps do not alter the color impression of the image.
  • the controlling of the illumination of the object and the exposure of the sensors is provided from signals in the processing and controlling unit 300.
  • the two broadband light sources can be incandescent lamps, gas lamps (like Hg, Xe, or mixtures), light emitting diodes (LEDs), or any other broadband light source. LED sources can be switched on and off at a high frequency rate, with rise and fall times faster than 100 microseconds.
  • the system can illuminate the object with alternating phases at video rate, i.e. approximately at 25 fps. At this and at higher illumination rates the visual perception of the illumination field is uniform, where any flickering effect is hardly observable. Additionally, since the two phases have complementary spectral illumination the overall color balance of the system is of a broadband white color, similar to the color appearance of the light of each of the broadband sources without filtering.
  • the sensor is preferably a multi-channel (multi color) sensor that has the capability to record the images in multiple spectral areas. Each spectral area has a distinct spectral sensitivity and records the reflected light of a spectral multiplex of various reflecting and fluorescence substances in the object. Examples of a multichannel color sensors arrays are the RGB (red- green-blue) or the CMYG (cyan-magenta-yellow-green) pattern sensors and typical color sensitivities are shown in fig. 6. These signals are spectrally mixed and are spectrally unmixed by image processing.
  • RGB red- green-blue
  • CMYG cyan-magenta-yellow-green
  • Images that are acquired are transferred to the processing unit 300 for a series of image processing operations, such as demonstrating, registration, noise filtering, background dark noise subtraction, color correction for the color frames, and spectral unmixing.
  • the spectral unmixing in the simplest form can be a linear transformation between the color channel images generated from the camera and the component space.
  • Components can be anything that the light can carry information from, such as materials, concentrations or properties, or quantities that can be derivatives from those components and they may have a particular spatial distribution similar to the parts 401, 402 of the object 400, and so on.
  • After the calculation of the images of the spatial distribution of the components 501, 502, and so on they can be stored, displayed, or overlaid on other images, with the use of colormaps, such as pseudocolor.
  • a) Absorber distribution The spectrum of the reflected light is shaped by the absorption and transmission spectrum in tissue, and this is recorded in the color sensor signals.
  • concentration of intrinsic tissue absorbers like oxygenated and deoxygenated hemoglobin, melanin, etc. or also externally administered absorption contrast agents e.g. methylene blue
  • c) Fluorochrome distribution Fluorescence comes either from endogenous fluorochromes or externally administered fluorescent contrast agents. The fluorescence signals are recorded by the color sensor and by system and tissue modeling and/or system calibration it is possible to derive the fluorochrome distribution. Additionally, it is possible to calculate ratios between fluorochrome maps, which convey more specific information on cancer.
  • f is the spectral fluorescence channel intensity density. It depends on the wavelength and the fluorescence channel.
  • Each fluorescence channel is characterized by a specific spectral light characteristic. I n the simplest case the spectral light characteristic of a fluorescence channel of the imaging system corresponds to a fluorophore. I n this case the (A, f) corresponds to the spectral emission spectrum of the fluorophore.
  • ⁇ ⁇ ( ⁇ , ) can be determined considering the fluorophore concentration, the fluorophores quantum yield and the spectral illumination light intensity.
  • ⁇ ( ⁇ , c) is the total transmission characteristics of the specific spatial color sensor or pixel which also exhibits the transmission characteristics of the optical system including the emission filter.
  • c(f) is the concentration of fluorophore f.
  • c(f) is the intensity factor.
  • Symbol for the concentration c is the same as the color channel index.
  • ⁇ ⁇ (1, ) is the molar spectral fluorescence intensity density describes the spectral profile of the emission of a fluorophore f. The intensity is scaled by the concentration of the fluorophore c( ) .
  • ⁇ ⁇ (1, ) is the normalized spectral reflectance intensity of a channel with a spectral distribution.
  • ⁇ ( ⁇ , f) could be the spectral response of the red receptor in the eye. This would lead to a natural color impression for this red channel.
  • the fluorescence intensities can be obtained by inverting the coefficient matrix M :
  • the equation system can be solved as a linear system of equations.
  • the variables S c are measured by the imaging system.
  • the values of c(f) can be calculated if the other
  • c(f) the matrix M(f, c) needs to be inverted. If the number of measured channels is larger than the number of fluorescence channels, the system is over-determined.
  • One option to handle this favorable situation is to compute the pseudo-inverse of M(f, c) which is not anymore a square matrix.
  • Various algorithms may be used to improve the outcome of the calculation and for example minimize noise originating from the measurements in the sensors.
  • the matrix M can be either calculated from system modeling and/or from system calibration.
  • system modeling the light path spectral content ca n be modeled from the light source to the color sensor array pixels.
  • Parameters include but are not limited to illumination source spectral distribution, the spectral transmission of the excitation filters, or the spectral profile of the illumination lights, the fluorochrome, excitation and emission spectra and the quantum yield, possibly the approximate depth of the components in tissue, also as the optical properties of tissue, the transmission characteristics of the imaging system (lenses, beam splitters, filters, mirrors, etc.) and finally the spectral sensitivities of the sensor array.
  • the modeling calculates the matrix M that connects the concentration information to the recorded signals (forward problem) and the component distribution can be derived from the solution of the inverse problem.
  • system calibration can be done with either recording the signals of components of known composition, concentration and location, and then solving for the unknown matrix M, or by a blind decomposition with unmixing algorithms, such as Principle Component Analysis (PCA), Independent Component Analysis (ICA), or similar statistical algorithms.
  • PCA Principle Component Analysis
  • ICA Independent Component Analysis
  • modeling, or in general the use of prior information can potentially be used to determine more unknowns than the number of measured channels.
  • the system can be modeled in more detail using a non-linear description.
  • non-linearities such as the detector or the quenching effect of high fluorochrome
  • the number of components unmixed is related to the total number of channels (colors) available from the combined images from the color sensors.
  • the number of spectral bands in the illumination and/or the transmission is independent from the number of channels (colors) and the number of components unmixed.
  • the more bands available in the region of interest the less likely is that a spectral feature from a particular component will not be recorded.
  • many "narrow" spectral bands offer more accurate color representation of the reflectance image, and more accurate unmixing of the various components.
  • spectral unmixing of various components is feasible with a number of spectral bands that is smaller than the number of channels.
  • the most basic light source 100 (see fig. 8) consists of two separate light sources 111, 121, which are preferably broadband LEDs 111 and 121. LEDs generally have very fast switching performance compared to conventional light sources such as Halogen or Xenon lamps.
  • the beams are optionally collimated with a collimation lens 119 and 129, and the source 121 is filtered with filter 122 and then combined using a polychroic mirror 101.
  • Figure 9A shows the spectrum of the broadband LEDs, that can be the same or different for light source 111, 121. The spectrum is typical for a white light LED.
  • Fig. 9B shows the transmission spectrum of the multi-band excitation filter 122.
  • Fig. 9C provides an intensity spectrum of the light emitted by LED source 121 and filtered by filter 122.
  • the emission spectrum of the two broadband high power LED sources with a maximum spectral power density is more than 30mW/nm.
  • This light is filtered with a multi-bandpass filter as shown in fig. 9B.
  • the filter has transmission bands (420 - 460 nm, 510.5 - 531.5 nm, 590 - 624 nm, 677.5 - 722.5 nm) with an approximate maximum transmission of 90% in each transmission band.
  • the attenuation characteristics of the filter in the blocking regions are typically at least of optical density 2 (O.D. 2). Usually the out of band rejection/attenuation characteristics of the filters are as good as O.D. 6.
  • the effective emission of the light source after filtering with the respective multi-band filter is illustrated in fig 9C.
  • the spectrum of the first light (source) is shaped by the accumulated light during this illumination phase and the spectrum of the second light (source) is the intrinsic broad-band emission profile such as in fig. 9. A or similar broadband.
  • all the drawn spectra of light are accumulated spectra during the respective phase.
  • the illumination field might not be optimal for the visual perception of an observer both in terms of intensity and of spectral content.
  • the two lights have different overall intensity and spectral content and when they are alternating may present a visual flickering of intensity or color. Additionally the spectral content is not balanced and the color appearance may not be natural.
  • An alternative illumination source is a variation of the basic light source, with the difference being that the second light is also filtered with a filter 112 (fig 10).
  • the basic advantage of filtering the second light is that it facilitates the optimization of the overall color perception and minimizes the flickering.
  • the filter 112 may also be a multiple bandpass filter. Its spectral transmission characteristics may be complementary to the filter 122 and may have the same or similar transmission characteristics to the fluorescence emission filter 213 in front of the sensor array 211 (see fig. 10B and fig. IOC).
  • the complementary filters 112 and 122 accumulatively provide a spectrally continuous illumination that is almost equal to the broadband illumination of the original broadband source thus providing natural color perception. Additionally the effect of intensity or color flickering is less. Nevertheless, the spectral shape of the light illumination of the second light (phase 2) is free to modify in order to achieve optimum color perception and minimal intensity flickering.
  • the output of the light source 100 can be coupled with a fiber coupling lens system into a light guide.
  • This light guide can either be a single optical fiber, a fiber bundle, or a liquid light guide.
  • one or more of the broadband light sources that are filtered with the multiband filters is replaced with a set of narrowband individually controlled sources optionally filtered by respective narrow band filters.
  • Such sources can be lasers, laser diodes, LEDs, etc.
  • a basic schematic is shown in fig. 11. where the light emitting module 111 has been replaced by multiple laser sources 123, 133, 143, 153. The emitted light of the module 123 is filtered by the filter 125.
  • the polychroic mirror 101 combines the radiation of the lasers with the radiation of the module 123. All the lights is coupled together to the fiber 103.
  • the beam splitter 101 may also be a polarization beam splitter. In this way the different sources at the same wavelength can be combined minimizing the losses.
  • Multiple lasers 133, 143 and 153 may replace one broadband source, e.g source 111 in Fig. 10.
  • the lasers may have a thin spectral emission profile or might also be tunable. Some lasers may require a cleanup filter to suppress unwanted amplified spontaneous emission.
  • the lasers may also be tunable in wavelength and intensity, they may be continuous wave or pulsed.
  • the different laser sources are combined by longpass polychroic mirrors 104 (cutoff wavelength) 415nm, 105 (cutoff wavelength 650nm) and 106 (plain mirror with high reflectivity around 785nm).
  • narrowband sources comprising the illumination in one phase may illuminate simultaneously, with full or partial time overlap, or may operate sequentially. Nevertheless, any time combination within the exposure period associated with an illumination phase is considered as an accumulative light spectral distribution in one illumination phase.
  • Fig.12 shows the spectrum of the first light source shown in fig.11.
  • the upper graph shows the individual spectral intensities of the broadband LED light source and the intensity of the individual narrowband lasers.
  • the middle graph shows the transmission spectrum of the emission filter.
  • the lower image shows the combined intensity of the light which is coupled into the optical fiber.
  • a broadband LED source covers the entire spectral range and is combined with narrowband laser sources which are may preferably be (for switching reasons) laser diodes.
  • narrowband laser sources which are may preferably be (for switching reasons) laser diodes.
  • popular modules like the 405 nm, the 633 nm and the 785 nm laser diode modules are used.
  • the diode laser at 405 nm can excite protoporphyrin IX (PPIX) which is widely used for brain surgery.
  • PPIX protoporphyrin IX
  • the diode laser at 633 nm can excite a highly stable and bright fluorophore such as Alexa647 used in fluorescent probes, and the diode laser emitting at 785 nm excites the clinically relevant indocyanine green (ICG).
  • ICG indocyanine green
  • the illumination lights are generated with several LED light sources as shown in fig. 13.
  • this option uses multiple LEDs (121, 131, 141, 151... ) that have a narrower spectral emission.
  • This requires a more complicated lightning device, but on the other hand the output power can be increased dramatically and the intensity of the different LEDs can be balanced independently.
  • Most monochrome LEDs still have a narrow emission with tails on the side spectrum.
  • excitation filters 122 132 142 152 may be optionally used in front of each LED to clean up the excitation spectra.
  • the LED comprised of many narrowband sources are regarded as one illumination light and the LEDs can illuminate simultaneously, with full or partial time overlap, or may operate sequentially with no overlap. Nevertheless, any time combination within the exposure period associated with an illumination phase is considered as an accumulative light spectral distribution in one illumination phase.
  • Such illumination sub-systems like the one described in fig. 13 can be combined in a multiphase illumination system as shown in the schematic of fig 14. Therein, two light sources 100 c and lOOd are provided, each coupling its emitted light into fibers 103c and 103d, respectively for illumination of sample 400.
  • Figs. 15 to 18 show each an emission spectrum of an LED light source, a transmission spectrum of a filter arranged in the emitted beam and an intensity spectrum of the emitted light after passing said filter. All four light sources together may replace one spectrally broadband light source.
  • This preferred configuration has one excitation LED for each band of the multi-band filters. This would require 8 single different LEDs for quadruple band-pass filters.
  • the spectra of such a configuration are shown in fig. 19.
  • Fig. 19 shows on the left side the spectrum of 4 LEDs, which constitute the first light source, the transmission spectrum of the corresponding filter and the resulting emission spectrum of the first light. On the right, corresponding spectra for the second light are shown.
  • each of the LEDs is associated with one light and thus with one phase.
  • the set of 4 individual LEDs can also be filtered using 4 individual single bandpass filters in front of each individual LED. Also the individual LEDs do not be strictly connected to one of the phases.
  • Fig. 20 shows the real emission spectra of 4 commercially available LEDs with emission maxima at 380 nm, 490 nm, 565 nm, and 635.
  • Fig. 21 shows the resulting illumination spectral profile of the above four LEDs filtered by a quadruple band pass filter.
  • the LED intensity is normalized for each of the 4 individual LEDs.
  • the first LED (380) marginally fits to the single band emission filter
  • the second and third LEDs (490 and 565) fit very well to their respective bandpass filter.
  • the LED 4 (635) does not fit very well to the spectral transmission of its excitation bandpass filter.
  • temporal switching between different light sources is performed using optical elements 168 and 165 with variable transparency.
  • these elements 168 and 165 with variable transparency are mechanical shutters. They can also be light modulators or acousto-optical devices.
  • the broadband light emanating from a light source 161 is split by a polychroic mirror 162, then filtered by complementary excitation filters 164 and 167 and merged again by a polychroic element 169 similar to element 162.
  • Mirrors 163 and 166 are used to align and guide the partial beam filtered by filter 165 in the system.
  • the excitation light should be collimated to minimize losses and optimize filter performance in the system. [Illuminating through the optical detection system]
  • the illumination system is configured to illuminate through the optical system.
  • An optical light guide delivers the light from the multispectral illumination system 100 into a part of the imaging device 200 at a connector port 204.
  • the illumination path may contain an optical lens system 203 to optimize the illumination on the object 400.
  • the light is then filtered by a polarization filter 206 and subsequently combined with the imaging path with a beam-splitter device 205.
  • a beam-splitter device 205 can be a polarization beam splitter cube 205.
  • the light is then passed through a rotatable half wave plate 207 which is rotating the angle of the polarization when light is passing through. This allows to reduce or eliminate reflections of reflected light depending on the position of the half wave plate.
  • the half wave plate 207 is located in front of the objective lens 201.
  • An alternative detection embodiment 200a is consisted of a cube beam splitter 202a instead of a mirror/dichroic mirror/polychroic mirror to split the beam into two separate beams as shown in fig. 24.
  • Cube beam splitters do not produce secondary images due to reflections on the non-coated side, and can in general have better optical flatness characteristics.
  • the splitting element 202a has preferably a spectral transmission characteristic according to the emission filters 223 and 213.
  • the beam splitter can also be a neutral cube beam splitter (preferably a 50:50 beam splitter) or a polarization beam splitter.
  • fig. 25 there are multiple sensor arrays (more than two, 211, 221, 231...), where more than one beam splitters (202b, 202c,..) are used to divide the beam into different imaging paths.
  • the filters 213, 223, 233 have complementary characteristics and are designed to support multiple detectors for multiphase operation of the system described later.
  • the two sensors are replaced by one single sensor array with larger area (see fig. 26).
  • the optical path is preferably split by the dichroic mirror 202d and then a first image is formed on first half of the sensor 211 through the imaging path 210.
  • the second image is aligned by the mirror 202f and then also imaged on the other half of the same sensor, but it is passing through an emission filter 223 complementary to the filter 213. In this way one sensor array detects two images.
  • This setup uses a three-way beam splitting to split the beam to individual sensors (fig. 28).
  • This setup has preferably the same first lens element 201 as the setup 200a (Fig. 24).
  • the beam splitting element 202e is different from 202a. It splits the beam into 3 different paths with a concept which is usually used for 3-CCD cameras. Instead of having 2 sensor paths, this setup uses 3 sensor paths 210, 220 and 230. Each path has its specific emission filter 213, 223 and 233 suitable for multiphase imaging to the sensors 211, 221 and 231. This approach can be extended to similar multiple beam splitters that offer multiple imaging paths (3 and more).
  • detector 200h uses two completely separate optical detection paths and no beam splitting compared to 200a. Each path consist of an objective lens element 212 and 222, an emission filter 213 and 223, optionally an imaging adapter lens 214 and 224 and the sensors 211 and 221.
  • Registration of the images is done at a later image processing step.
  • the sensors described in the previous embodiments are in general multichannel color sensors. This means that each individual sensor records the light field in multiple distinct spectral distributions. This can be achieved with various options: a) sensors that have microfilters in front of the pixels following the Bayer RGGB microfilter pattern or modifications of this like the RG(IR)B, the CMYG, b) any other filter mosaic patterns where each pixel records lighter with a distinct spectral distribution, and/or c) any further beam splitting, color filtering and imaging on monochrome sensors.
  • the RGGB pattern achieves more accurate color reproduction, while the CMYG can be more sensitive (fig 30).
  • the full resolution color image can be retrieved by demosaicing, which can take place in the camera hardware, or later in image processing.
  • the microfilter pattern can in general be extended to multiple colors or spectral transmission profiles like ABCDEFGHI etc. An example like this is the lithographically patterned dichroic filter arrays as disclosed in US 6638668 B2.
  • the multichannel color sensors can be based on Foveon X3 sensors [see US 6632701] or similar technologies (Fig. 31).
  • the Foveon sensor is having photosensors spatially arranged in x- and y direction and that multiple layers (layer 1, layer 2,...) are vertically stacked. Each layer is sensitive to different spectral areas due to the silicon absorption and the different transmission depths for the layer above light, thus the images generated of each layer corresponds to different color. With this it is possible to achieve higher spatial resolution.
  • the multichannel sensors like 211 or 221 are replaced by a monochrome sensor 251 that the beam is split in three parts with the use of beam splitters/or mirrors 252 and 253 and filtered with filters or with dichroic mirrors (fig. 32).
  • Each filter or dichroic mirror has a particular transmission spectral profile, that separates the light to the different colors in fig. 32B and C. Different images are formed in the monochrome sensor each imaging a different spectral band.
  • a multiple color channel can be implemented with multiple light splitting and filters, such as the prism 3-CCD geometry (as disclosed in U.S. 3659918).
  • each path is filtered to carry light with the spectrum of the specific color, for example RGB.
  • This approach can be extended to similar multiple beam splitters that offer multiple imaging paths (3 and more).
  • ambient light For most fluorescence applications ambient light needs to be avoided or blocked because its intensity is several orders of magnitude stronger than the intensity of the fluorescence light emerging from the fluorescent dye.
  • Ambient light might come from the sun and pass through the windows onto the object or it might be emitted by the room lights.
  • the environment In current state-of-the- art systems usually, the environment is dark to avoid the intensive signal from ambient light in the fluorescence channels.
  • the specific wavelength regions of ambient light, which would pass the emission filter may be blocked by filters. Unfortunately such filters are usually very expensive and it is not possible to cover big windows or room lights with such filters or they are just not available for any configuration.
  • the second method/embodiment uses the dead times of the sensor array in between frames for ambient illumination.
  • the illumination of the room lights are pulsed at a high frequency compared to maximum frequency perception of the human eye (for example at 200 Hz).
  • the duration (duty cycle) of the pulses is typically a small fraction of the whole period (for example 5-20% of the period, typically 0.1-5 ms) as this allows longer exposure time for the fluorescence imaging (see fig. 33).
  • the light path for imaging fluorescence signals is blocked during the pulses of light of the ambient illumination.
  • the figure shows the phases of the imaging system and the respective timing of the shutter device to allow ambient illumination.
  • a room illumination/ambient illumination light source 902 is provided, the light of which is coupled into the excitation light path.
  • an additional shutter 900 is provided in the imaging path.
  • the shutter is placed in front of the objective lens of the optical system 200 for simplicity reasons. Nevertheless it can also be placed at another position in the path.
  • the shutter device 900 can be included in the imaging path directly in front of the sensor arrays. Both, the shutter 900 and the room illumination 902 are controlled from the control/processing unit 300.
  • the frequency of operation of the ambient illumination from source 902 is not necessarily connected to the frequency of operation of the fluorescence imaging system. It is preferable if the imaging system runs at 30-60Hz to generate fluent stream of images of fluorescence and reflectance for the human eye.
  • the ambient illumination 902 is preferably operated with a frequency which is higher so the human eye does not perceive any flickering in the room environment.
  • the frequency of operation of the ambient lighting system 902 is a higher harmonic of the frequency of the imaging.
  • each sequentially taken picture is equally influenced by the closed imaging path. But it would also be possible to detect the ambient illumination timing and digitally correct for the influence of the slightly differently shuttered imaging path if necessary.
  • the shutter 900 can be any electromechanical device that can allow or block light from propagation along the beam path.
  • the ambient light and the optical imaging path 903 is shuttered by a beam chopper wheel 901 (see fig. 35) rotating at half the frequency of the shutter effect.
  • Chopper wheels 901 are a good choice to interrupt imaging paths with a certain frequency and usually operate at higher frequencies compared to optical shutters.
  • a chopper wheel can be exchanged by different devices like electro optical modulator, SLMs, or acousto-optical modulators to hold the recording of the image by making the path opaque.
  • the path is closed using polarization filters and using electronic devices with a variable polarization sensitive transmission of light. This also allows to effectively block the imaging path.
  • the light source can be any type of ambient light source that can operate with short pulses.
  • the light source 902 preferably consists of electronically pulsed LEDs. Such LEDs are well suitable for the ambient illumination of an operation theater and can be pulsed at a very high frequency compared to the frequency of the human eye.
  • An alternative embodiment as shown in fig. 36 uses an additional phase (3rd phase) of illuminating light from a different light source always between the phases 1 and 2 of the imaging setup for ambient illumination.. This phase runs at double the frequency of the other phases.
  • the light source can either be independent similar to the light source 902 or be included in the light source 100 of the illumination system.
  • the light emitted by this light source is not necessarily used for imaging, but may mainly be used to improve the visual perception for the human eye in the object and/or the surrounding environment.
  • the illumination of the imaging area is optimized only for the detection of image components and the image processing, and in particular for the unmixing of the different fluorophores.
  • an illumination is not optimal for the visual impression for a surgeon and may result a low image contrast and non-natural visual impression.
  • the spectral distribution and intensity of the additional third illumination phase however is free to optimize the overall visual perception and brightness for the users (surgeon and medical personnel in the OR) as perceived accumulatively for all illumination phases.
  • the illumination pulses in the 3rd phase are short enough to fit in the dead time of the imaging sensors between the two phases (see fig 36) Usually the dead time occurs when transferring the data from the sensor 200 to the controlling unit 300. Thus short pulses with a high accuracy are required.
  • the imaging system works at a frequency of 30 Hz, the pulsed ambient illumination can work at double the frequency, i.e. 60 Hz. If we assume that the ambient illumination should just consume a duty cycle of 1%, the pulse width of the pulses should be in the order of 170 ⁇ . If the ambient illumination consumes 5% duty cycle, the additional illumination phase provides a brighter field, and the pulsed ambient illumination duration is 800 ⁇ .
  • the same concept can be extended to further cameras and phases in more elaborate imaging scenarios. For example extent to 3 cameras and 3 phases, 4 cameras and 4 phases and so on. These allow for example to acquire additional spectral information on both the reflection and fluorescence images. Additionally, an alternative configuration operates in two phases, but may use more than two cameras, which offers an advantage in case two multi-channel cameras cannot resolve essential features like fluorescence lifetime or sensitivity in the infrared region.
  • This example system has three different light sources 110, 120 and 130.
  • the respective example emission spectra of the light sources are drawn in fig. 38, 39 and 40, respectively.
  • the sample is illuminated in 3 phases by the light sources. In the first phase, the first light source illuminates the sample, in the second phase the second light source illuminates the sample and in the third phase the third light source illuminates the sample.
  • the detector consists of 3 separate sensors 211 221 and 231 and the light is attenuated with the filters 213, 223, and 233 placed in front of each sensor respectively.
  • Each sensor has a respective transmission filter that matches the emitted spectrum in of each respective light source placed in the imaging path, i.e. filter with the transmission characteristics shown in fig. 41 is placed in front of sensor 1, filter shown in fig. 42 in front of sensor 2, and filter shown in fig. 43 in front of sensor 3. In this way there is a correspondence between the illumination spectra of each source to the light reaching each sensor. In each phase, all three sensors record a multi-channel (color) image.
  • table 1 the correspondence of illumination lights and filters to transmit or attenuate the reflected and emitted light is shown.
  • phase 1 sensor 1 records a reflection image in the spectral bands in which are illuminated by the light source 1.
  • camera 2 and camera 3 cannot record the light emitted from the light source (source 1), as the filters placed in their imaging paths block the excitation light. But during this phase, each of these cameras records the fluorescence emission in the respective filter transmission bands.
  • the second and third phase one camera detects the reflected excitation light and the other two the emitted fluorescence, as shown in the table 2.
  • a full imaging cycle closes in 3 phases as shown in fig.
  • the recording of all color channels of all cameras in each phase results in a combined data set (or a combined image) comprised of multiplexed spectral images, that each has different information, either of reflected excitation or fluorescence emission.
  • the integration time of the each camera can vary in each phase to accommodate for the dynamic range of each image. Table 2. camera function for each phase
  • each sensor has 3 detection channels (for example a standard RGB camera)
  • the system records combined reflectance images from 9 channels and combined fluorescence information from 18 channels.
  • the major advantage of this setup is the increase in the total number of channels for fluorescence and reflection images, which enables the more accurate decomposition of the reflectance and fluorescence images to their accurate components after images processing.
  • the filter design and manufacturing becomes more complicated, the image processing becomes more computationally intensive, and the total light reaching each sensor is less.
  • the resulting lower light intensity and there for the lower S/N ratio can be compensated by longer exposure times, more sensitive camera sensors, and higher intensity light sources.
  • the principle of time and spectral multiplexing can also be extended to 4 phases, 4 light sources and 4 cameras as shown in fig. 45.
  • the sample is illuminated with light source 110 in phase 1, with light source 120 in phase 2, with light source 130 in phase 3 and with light source 140 in phase 4.
  • the 4 different illumination spectra of the light sources are illustrated in figures 46, 47, 48 and 49.
  • the characteristics of the different filters 213, 223, 233, and 243 in front of the cameras which are specified in table 3 with respect to the spectral
  • transmitted means that is a rather large part of the intensity is actually transmitted while blocked means that the light intensity is attenuated by the filter preferably by a factor of 10 ⁇ -4 to 10 ⁇ -6.
  • Table 3 Filter design for the different cameras with respect to the illumination spectrum of the different light sources
  • the sample is imaged splitting the imaging path into 4 partial paths and in each path the light is filtered and then focused onto the respective cameras.
  • the filters in front of each camera transmit the light, which is emitted by the light source of the same number, but blocks all the light emitted from the other 3 light sources.
  • Each camera records in one phase a reflection image and in the 3 other phases a fluorescence image.
  • the table 4 shows the recording combinations.
  • Table 4 Recording of fluorescence and reflection images for the different cameras in different phases
  • a full imaging cycle closes in 4 phases and in total, the recording of all color channels of all cameras in each phase results in a combined data set (or a combined image) comprised of multiplexed spectral images, that has different information, either of reflected excitation or fluorescence emission.
  • the integration time of the each camera can vary in each phase to accommodate for the dynamic range of each image.
  • a combined plot with all the transmission spectra for the four phases is shown in fig. 55.
  • This concept can be extended to a higher number of cameras and phases according to the shown principle.
  • the multispectral imaging method and system can be implemented by integrating into various imaging instruments, e. g. integrated in medical instruments.
  • the multispectral imaging system is used with a zoom lens 291 as an objective lens by attaching the detector 200 with a camera adaptor, where the illumination system 100 delivers the light to the object with a light guide.
  • the detection system 200 is connected to the video port of a surgical microscope 292 and the illumination system 100 is connected with a light guide to the illumination port to illuminate the object through the objective lens of the microscope.
  • the detection system 200 is connected to the eyepiece port of a rigid endoscope optionally with the use of an adaptor and the illumination system 100 is connected with a light guide to the illumination port.
  • the detection system 200 is miniaturized and integrated into the tip of a flexible endoscope, while the illumination system 100 is attached to the illumination port of the endoscope.
  • the detection system 200 is connected to the camera port of the flexible fiberscope, which transfers the image from the tip to the distal point with the use of a flexible fiber bundle, and the illumination system 100 is connected to the illumination port.
  • Imaging of blood oxygenation is imaged by assessing the relative concentration of oxygenated to de-oxygenated hemoglobin (HbO and Hb) on tissue. Since HbO and Hb have distinct absorption spectra as shown in fig. 57, the reflected light carries spectral profile information that can be recorded in the system. By spectrally unmixing the multiple reflectance components it is possible to generate a) an RGB image to be displayed in a visualization system and b) an additional map of the biodistribution of the HbO and Hb components. The oxygen saturation maps are calculated by the ratio between the HbO to the total hemoglobin saturation.
  • Detection of Cancer lesions, anatonimcal features, or functional conditions Another envisioned application is to use the system to visualize the biodistribution of injectable fluorescent contrast agents for in-vivo clinical diagnostic imaging.
  • These fluorescent contrast agents may be non-targeted, like Fluorescin or Indocyanin Green to highlight vascularization, blood perfusion etc., or targeted in a way that can highlight with fluorescence diseases, such as cancer, medical conditions, such as inflammation, or anatomical features, such as neures or lymph nodes, by binding to molecular sites associated to relative functional or pathological activity in tissue.
  • the chemical environment can influence the emission or the excitation of fluorescent dyes.
  • One of these parameters changing the dye absorption and emission characteristics is the pH value.
  • the excitation bands of the respective filters and light sources optimized in a way to detect signal which is spectrally sensitive to changes of the pH value. It is also preferable to have excitation bands so that some of the detected channel(s) depend maximally on the pH value, whereas other channel(s) are mostly insensitive to changes of the pH value.
  • the excitation filter bands should be adjusted such that the center of the respective bands either matches a spectral point where the dye excitation spectrum varies maximal on a change of pH value or on a spectral point where the dye excitation spectrum minimally depends on the pH value.
  • the recorded images are multi spectrally recorded, spectrally unmixed and processed in such a way that they visualize the spatial distribution of the pH values.
  • Application Scenario Distinguishing tumor infiltration zone and solid tumor mass by differences in the PPIX emission spectrum
  • PPIX protoporphyrin IX
  • 5-ALA is administered to the patient leading to an accumulation of protoporphyrin IX (PPIX) in tumor tissue.
  • PPIX protoporphyrin IX
  • the substance PPIX is both, a fluorescent dye and also an agent for photodynamic therapy.
  • the fluorescence emission spectrum of the PPIX varies depending on the location inside the tumor. More precisely the infiltration zone exhibits a different fluorescence emission spectrum compared to the solid tumor mass. This spectral difference can be used in order to differentiate between the tumor mass and the infiltration zone.
  • the PPIX is preferably excited in both phases at approximately 405nm. But in phase one, the emission is preferably recorded in a spectral band between 590nm to 627nm. In phase two, the fluorescence is preferably recorded in the spectral region between 628nm to 650nm.
  • tissue autofluorescence is the fluorescence usually emitted without administering fluorescent contrast agents e. g. fluorophores.
  • the tissue intrinsic autofluorescence is attributed to various molecules that exist or are produced in the tissues, such as NADPH, flavins, collagen, elastin, and others.
  • the existence, production, accumulation, or other concentration properties is linked to various tissue features, such as anatomical, functional, and pathological features.
  • the multispectral imaging of tissue autofluorescence and the spectral unmixing of the associated compounds according to the invention can reveal features or characteristics of tissue that aid the assessment or the diagnosis of a medical condition. Multispectral imaging and unmixing of the autofluorescence can take place together with systemically administered fluorescent molecules.
  • the retina can be imaged through the eye.
  • this imaging modality is used in clinical practice mainly for diagnostic purposes of the retina itself.
  • the eye provides a clear window to the blood vessels of the body looking directly in the retinal vessels.
  • multispectral imaging of the retina and spectral unmixing according to the invention it is possible to identify fluorescent molecules that are either existing in the retina or circulate in its blood vessels. These fluorescent molecules may have been systemically administered, to freely circulate or to target cells (possibly metastatic cancer cells), microorganisms, viruses, or molecules. Multispectral imaging and unmixing can identify these substances, which can provide information about the blood circulation in general, or the circulation of the targets, that can help to assess the functional, or pathological condition of the "patient". Therefore it is possible to use retina imaging to obtain information about the retina itself and also to obtain information about compounds circulating in the blood.
  • Robotic surgery Robotic surgery
  • An interesting application of the multispectral imaging and system is to combine it with a surgical robotic system.
  • it can provide the surgeon that operates with visual multispectral information either in the reflectance color domain, or in the (auto-)fluorescence domain, about tissue anatomy, function or disease.
  • a second level can provide input that increases the safety of the robot operation, for example prohibiting the doctor from accidentally damaging (i.e. cutting) tissue (for example, nerves).
  • tissue for example, nerves
  • a third level it can directly provide input and or feedback to an automated robotic surgery procedure that has reduced or minimum human controlling.
  • the described scenarios have the same number of phases (light sources) and sensors.
  • the principle of a system which uses combined temporal and spectral multiplexing also cover embodiments which have a different number of phases than cameras. Subsequently two different embodiments are described as examples. The first of those two scenarios has more cameras than phases (lights), while the second of those scenarios has more phases (lights) than cameras.
  • the embodiment which is described here with reference to fig. 58 is derived from the basic embodiment shown in figure 3. It has one additional camera in the multispectral detector 200. This detector can record light also in both phases (from both lights). The additional sensor serves for example to record light in a spectral region, in which the other sensors are not sensitive, for example in the UV or more likely in the infrared region.
  • this additional monochrome sensor is high in the near infrared (NIR) region above 800nm. So this sensor supports the other sensors by recording the light for example of a fluorescent dye emitting in the NIR such as ICG. Such dyes are desirable for clinical applications because they show less absorption in tissue by hemoglobin than dyes emitting in the blue or green region. Additionally this sensor can also record a reflection image in one of the two phases. The system operates in two phases and thus has two lights. The lights need to match the spectral demands in the NIR region to provide excitation light for the fluorescent dye and also the provide illumination for reflection light to be recorded in the NIR region.
  • NIR near infrared
  • This part describes an embodiment which has more phases (lights) than cameras.
  • the embodiment is derived from the basic embodiment, shown in figure 3. It has an additional phase and thus an additional third light. Thus one period of the imaging system is divided into three different acquisition phases.
  • This embodiment has additional capabilities to distinguish between dyes which have very similar emission spectra but very different excitation spectra.
  • excitation scanning can help to distinguish between the dyes.
  • the emission of the two dyes in the same spectral region is recorded in two phases. But the excitation in these two phases needs to be different in order to be able to distinguish between the different dyes.

Landscapes

  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Pathology (AREA)
  • Surgery (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Chemical & Material Sciences (AREA)
  • Public Health (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Veterinary Medicine (AREA)
  • Biophysics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Immunology (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • Optics & Photonics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Chemical Kinetics & Catalysis (AREA)
  • Plasma & Fusion (AREA)
  • Ophthalmology & Optometry (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
  • Instruments For Viewing The Inside Of Hollow Bodies (AREA)
  • Microscoopes, Condenser (AREA)
  • Endoscopes (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)

Abstract

La présente invention concerne l'imagerie multispectrale d'échantillons, en particulier de tissus biologiques. L'invention concerne également un procédé d'acquisition d'images de fluorescence et de réflexion d'un objet (400) comprenant les étapes consistant à éclairer alternativement l'objet (400) à l'aide d'au moins une première lumière et d'une seconde lumière, la première lumière et la seconde lumière étant spectralement telles à ce qu'au moins une lumière comprenne plusieurs régions spectrales de haute intensité lumineuse séparées par une ou plusieurs régions spectrales de faible intensité lumineuse, les régions spectrales de la première lumière et de la seconde lumière à intensité élevée ne se chevauchent pas, au moins partiellement, et au moins l'une des deux lumières présentant au moins une région de faible intensité lumineuse dont la longueur d'onde est supérieure à celle de la région voisine d'intensité lumineuse élevée, et à enregistrer au moins une première image de l'objet (400) et une seconde image de l'objet (400) tout en éclairant l'objet (400) à l'aide d'au moins l'une des lumières, la lumière à enregistrer en tant que première image étant modifiée de telle sorte qu'au moins une région spectrale d'intensité élevée de la seconde lumière soit atténuée et la lumière à enregistrer en tant que seconde image étant modifiée de telle sorte qu'au moins une région spectrale d'intensité élevée de la première lumière soit atténuée.
EP15730081.5A 2014-06-05 2015-06-03 Procédé et moyen d'imagerie multispectrale Active EP3152549B1 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
EP14171378 2014-06-05
EP15160630 2015-03-24
PCT/EP2015/062448 WO2015185662A2 (fr) 2014-06-05 2015-06-03 Procédé et moyen d'imagerie multispectrale

Publications (3)

Publication Number Publication Date
EP3152549A2 true EP3152549A2 (fr) 2017-04-12
EP3152549B1 EP3152549B1 (fr) 2023-06-07
EP3152549C0 EP3152549C0 (fr) 2023-06-07

Family

ID=53269508

Family Applications (3)

Application Number Title Priority Date Filing Date
EP15730081.5A Active EP3152549B1 (fr) 2014-06-05 2015-06-03 Procédé et moyen d'imagerie multispectrale
EP15725386.5A Active EP3152548B1 (fr) 2014-06-05 2015-06-03 Procédés et moyens d'imagerie multispectrale
EP21174170.7A Active EP3940371B1 (fr) 2014-06-05 2015-06-03 Procédé et appareil d'imagerie pour l'acquisition d'images de fluorescence et de réflectance

Family Applications After (2)

Application Number Title Priority Date Filing Date
EP15725386.5A Active EP3152548B1 (fr) 2014-06-05 2015-06-03 Procédés et moyens d'imagerie multispectrale
EP21174170.7A Active EP3940371B1 (fr) 2014-06-05 2015-06-03 Procédé et appareil d'imagerie pour l'acquisition d'images de fluorescence et de réflectance

Country Status (9)

Country Link
US (2) US10481095B2 (fr)
EP (3) EP3152549B1 (fr)
JP (2) JP6728070B2 (fr)
CN (2) CN106572792B (fr)
CA (2) CA2951075A1 (fr)
DK (1) DK3940371T3 (fr)
ES (2) ES2883220T3 (fr)
FI (1) FI3940371T3 (fr)
WO (2) WO2015185661A1 (fr)

Families Citing this family (117)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8498695B2 (en) 2006-12-22 2013-07-30 Novadaq Technologies Inc. Imaging system with a single color image sensor for simultaneous fluorescence and color video endoscopy
CN102036599B (zh) 2008-03-18 2013-06-19 诺瓦达克技术公司 用于组合的全色反射和近红外成像的成像系统
US9674920B2 (en) * 2014-12-01 2017-06-06 Industrial Technology Research Institute Illumination system and method for developing target visual perception of an object
CA2998920A1 (fr) 2015-11-13 2017-05-18 Novadaq Technologies ULC Systemes et procedes d'eclairage et d'imagerie d'une cible
JP6671653B2 (ja) * 2015-12-11 2020-03-25 大学共同利用機関法人情報・システム研究機構 画像処理装置、画像処理方法及びプログラム
EP3190394A3 (fr) * 2015-12-15 2017-11-15 Applied Spectral Imaging Ltd. Systèmes et procédés d'imagerie spectrale
CA3009419A1 (fr) 2016-01-26 2017-08-03 Novadaq Technologies ULC Plateforme configurable
JP7030056B2 (ja) * 2016-01-28 2022-03-04 シーメンス・ヘルスケア・ダイアグノスティックス・インコーポレーテッド 試料容器と試料の特徴付けのための方法及び装置
JP2017131559A (ja) * 2016-01-29 2017-08-03 ソニー・オリンパスメディカルソリューションズ株式会社 医療用撮像装置、医療用画像取得システム及び内視鏡装置
USD916294S1 (en) 2016-04-28 2021-04-13 Stryker European Operations Limited Illumination and imaging device
US10869645B2 (en) 2016-06-14 2020-12-22 Stryker European Operations Limited Methods and systems for adaptive imaging for low light signal enhancement in medical visualization
TWI586957B (zh) * 2016-06-24 2017-06-11 諾貝爾生物有限公司 多通道螢光檢測系統及其方法
JP6570484B2 (ja) * 2016-07-04 2019-09-04 Hoya株式会社 分析装置
US10371626B2 (en) * 2016-08-17 2019-08-06 Kla-Tencor Corporation System and method for generating multi-channel tunable illumination from a broadband source
WO2018035597A1 (fr) * 2016-08-26 2018-03-01 Huron Technologies International Inc. Microscope à balayage utilisant un filtre de balayage en mosaïque
US10490160B2 (en) 2016-09-22 2019-11-26 Apple Inc. Light sensor with spectral sensing and color matching function channels
EP3330685A1 (fr) * 2016-12-05 2018-06-06 Sick Ag Dispositif de mesure de la spectroscopie d'absorption
DE102016125524A1 (de) 2016-12-22 2018-06-28 Arnold & Richter Cine Technik Gmbh & Co. Betriebs Kg Elektronisches Mikroskop
CA3049922A1 (fr) 2017-02-10 2018-08-16 Novadaq Technologies ULC Systemes et procedes d'imagerie a fluorescence portative a champ ouvert
JP6726859B2 (ja) * 2017-02-20 2020-07-22 パナソニックＩｐマネジメント株式会社 蛍光検出器及びその制御方法
JP6939000B2 (ja) * 2017-03-23 2021-09-22 株式会社Ｊｖｃケンウッド 撮像装置及び撮像方法
WO2018177944A1 (fr) * 2017-03-28 2018-10-04 Philips Lighting Holding B.V. Source de lumière et procédé pour augmenter la perception des couleurs chez les personnes souffrant d'une vision défectueuse des couleurs
JP6977287B2 (ja) * 2017-03-29 2021-12-08 東ソー株式会社 複数の蛍光物質を用いた標識方法
US11650157B2 (en) * 2017-04-20 2023-05-16 Irillic Pvt Ltd System for extending dynamic range and contrast of a video under fluorescence imaging
WO2018204835A1 (fr) * 2017-05-04 2018-11-08 June Brain Llc Système de surveillance cérébrale
DE102018204426B4 (de) * 2017-05-11 2024-02-29 Carl Zeiss Meditec Ag Mikroskopiesystem
CN108937849A (zh) * 2017-05-29 2018-12-07 王虎 一种用于肿瘤纳米靶向探针荧光成像及手术导航指示系统
CN107301639B (zh) * 2017-06-02 2019-10-29 清华大学 利用数字图像技术改善色度检测线性范围的方法以及装置
WO2018225122A1 (fr) * 2017-06-05 2018-12-13 オリンパス株式会社 Dispositif d'endoscope
EP3417763A1 (fr) * 2017-06-22 2018-12-26 Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH) Système d'imagerie endoscopique
DE102017117428A1 (de) * 2017-08-01 2019-02-07 Schölly Fiberoptic GmbH Bildgebendes Verfahren unter Ausnutzung von Fluoreszenz sowie zugehörige Bildaufnahmevorrichtung
DE102017121483B3 (de) 2017-09-15 2019-03-07 Leica Microsystems Cms Gmbh Verfahren zur Untersuchung einer multispektralen Probe, Steuereinheit hierfür und Mikroskop-Anordnung
CN111373413B (zh) * 2017-11-07 2021-11-09 眼验股份有限公司 用于产生图像的表示的方法和成像系统
JP7000933B2 (ja) 2017-12-27 2022-01-19 カシオ計算機株式会社 撮像装置及び撮像方法
IL275563B1 (en) * 2017-12-27 2024-03-01 Ethicon Llc Hyperspectral imaging with tracking with a device in a poor light environment
WO2019131586A1 (fr) * 2017-12-27 2019-07-04 カシオ計算機株式会社 Dispositif de capture d'image et procédé de capture d'image
WO2019164455A1 (fr) 2018-02-21 2019-08-29 Optotek D.O.O. Appareil et procédé de traitement d'images pour la détection d'irrégularités dans un tissu
US11382487B2 (en) * 2018-04-03 2022-07-12 Curadel, LLC Micro CMOS scopes for medical imaging
CN108956564B (zh) * 2018-06-21 2020-12-29 深圳市优迈医学科技有限公司 光敏剂浓度检测装置、系统以及方法
EP3814741A4 (fr) * 2018-06-28 2022-03-02 The Regents Of The University Of California Production d'une image composite d'un échantillon de tissu coloré par combinaison de données d'image
KR102599207B1 (ko) 2018-07-20 2023-12-15 삼성전자 주식회사 전자 디바이스의 표면 측정 장치 및 방법
WO2020039520A1 (fr) * 2018-08-22 2020-02-27 オリンパス株式会社 Dispositif de traitement d'image, système d'image, procédé et programme d'exploitation de dispositif de traitement d'image
EP3866462B1 (fr) * 2018-10-09 2022-11-02 FUJIFILM Corporation Dispositif d'imagerie
US11714004B2 (en) * 2018-11-15 2023-08-01 Ams Sensors Singapore Pte. Ltd. Measurements using systems having multiple spectrometers
CN113260835A (zh) 2018-12-14 2021-08-13 光谱Md公司 用于高精度多孔径光谱成像的系统和方法
EP3669743B1 (fr) * 2018-12-20 2024-04-03 Leica Instruments (Singapore) Pte. Ltd. Système et procédé, en particulier pour microscopes et endoscopes, pour la création d'une image à grande gamme dynamique d'un fluorophore fluorescent
JP2020130538A (ja) * 2019-02-18 2020-08-31 ソニー・オリンパスメディカルソリューションズ株式会社 医療用光源装置及び医療用観察システム
US11747205B2 (en) * 2019-02-27 2023-09-05 Deep Smart Light Ltd. Noninvasive, multispectral-fluorescence characterization of biological tissues with machine/deep learning
WO2020197497A1 (fr) * 2019-03-27 2020-10-01 Ams Sensors Singapore Pte. Ltd. Modules de capteurs spectraux à auto-étalonnage
US10878548B2 (en) 2019-03-28 2020-12-29 Advanced New Technologies Co., Ltd. Specular reflection reduction using polarized light sources
EP3958724A4 (fr) * 2019-04-26 2023-01-11 ChemImage Corporation Produit d'imagerie hybride et système endoscopique hybride
CN110236694B (zh) * 2019-06-04 2021-09-10 广东欧谱曼迪科技有限公司 基于光谱响应特性的同屏近红外双谱荧光成像方法及系统
US11671691B2 (en) 2019-06-20 2023-06-06 Cilag Gmbh International Image rotation in an endoscopic laser mapping imaging system
US11898909B2 (en) 2019-06-20 2024-02-13 Cilag Gmbh International Noise aware edge enhancement in a pulsed fluorescence imaging system
US11617541B2 (en) 2019-06-20 2023-04-04 Cilag Gmbh International Optical fiber waveguide in an endoscopic system for fluorescence imaging
US11754500B2 (en) 2019-06-20 2023-09-12 Cilag Gmbh International Minimizing image sensor input/output in a pulsed fluorescence imaging system
US11432706B2 (en) 2019-06-20 2022-09-06 Cilag Gmbh International Hyperspectral imaging with minimal area monolithic image sensor
US11925328B2 (en) 2019-06-20 2024-03-12 Cilag Gmbh International Noise aware edge enhancement in a pulsed hyperspectral imaging system
US11237270B2 (en) 2019-06-20 2022-02-01 Cilag Gmbh International Hyperspectral, fluorescence, and laser mapping imaging with fixed pattern noise cancellation
US11276148B2 (en) * 2019-06-20 2022-03-15 Cilag Gmbh International Super resolution and color motion artifact correction in a pulsed fluorescence imaging system
US11674848B2 (en) 2019-06-20 2023-06-13 Cilag Gmbh International Wide dynamic range using a monochrome image sensor for hyperspectral imaging
US11540696B2 (en) 2019-06-20 2023-01-03 Cilag Gmbh International Noise aware edge enhancement in a pulsed fluorescence imaging system
US11633089B2 (en) 2019-06-20 2023-04-25 Cilag Gmbh International Fluorescence imaging with minimal area monolithic image sensor
US11360028B2 (en) 2019-06-20 2022-06-14 Cilag Gmbh International Super resolution and color motion artifact correction in a pulsed hyperspectral, fluorescence, and laser mapping imaging system
US11716543B2 (en) 2019-06-20 2023-08-01 Cilag Gmbh International Wide dynamic range using a monochrome image sensor for fluorescence imaging
US11793399B2 (en) 2019-06-20 2023-10-24 Cilag Gmbh International Super resolution and color motion artifact correction in a pulsed hyperspectral imaging system
US11284783B2 (en) 2019-06-20 2022-03-29 Cilag Gmbh International Controlling integral energy of a laser pulse in a hyperspectral imaging system
US11288772B2 (en) 2019-06-20 2022-03-29 Cilag Gmbh International Super resolution and color motion artifact correction in a pulsed fluorescence imaging system
US11412920B2 (en) * 2019-06-20 2022-08-16 Cilag Gmbh International Speckle removal in a pulsed fluorescence imaging system
US11758256B2 (en) 2019-06-20 2023-09-12 Cilag Gmbh International Fluorescence imaging in a light deficient environment
US11931009B2 (en) 2019-06-20 2024-03-19 Cilag Gmbh International Offset illumination of a scene using multiple emitters in a hyperspectral imaging system
US11076747B2 (en) 2019-06-20 2021-08-03 Cilag Gmbh International Driving light emissions according to a jitter specification in a laser mapping imaging system
US11412152B2 (en) * 2019-06-20 2022-08-09 Cilag Gmbh International Speckle removal in a pulsed hyperspectral imaging system
US11457154B2 (en) 2019-06-20 2022-09-27 Cilag Gmbh International Speckle removal in a pulsed hyperspectral, fluorescence, and laser mapping imaging system
US12013496B2 (en) 2019-06-20 2024-06-18 Cilag Gmbh International Noise aware edge enhancement in a pulsed laser mapping imaging system
US11398011B2 (en) 2019-06-20 2022-07-26 Cilag Gmbh International Super resolution and color motion artifact correction in a pulsed laser mapping imaging system
US11471055B2 (en) 2019-06-20 2022-10-18 Cilag Gmbh International Noise aware edge enhancement in a pulsed fluorescence imaging system
US11903563B2 (en) 2019-06-20 2024-02-20 Cilag Gmbh International Offset illumination of a scene using multiple emitters in a fluorescence imaging system
US11389066B2 (en) 2019-06-20 2022-07-19 Cilag Gmbh International Noise aware edge enhancement in a pulsed hyperspectral, fluorescence, and laser mapping imaging system
US11533417B2 (en) 2019-06-20 2022-12-20 Cilag Gmbh International Laser scanning and tool tracking imaging in a light deficient environment
US10841504B1 (en) 2019-06-20 2020-11-17 Ethicon Llc Fluorescence imaging with minimal area monolithic image sensor
US11550057B2 (en) 2019-06-20 2023-01-10 Cilag Gmbh International Offset illumination of a scene using multiple emitters in a fluorescence imaging system
US11102400B2 (en) 2019-06-20 2021-08-24 Cilag Gmbh International Pulsed illumination in a fluorescence imaging system
US11187658B2 (en) 2019-06-20 2021-11-30 Cilag Gmbh International Fluorescence imaging with fixed pattern noise cancellation
US11589819B2 (en) 2019-06-20 2023-02-28 Cilag Gmbh International Offset illumination of a scene using multiple emitters in a laser mapping imaging system
US11622094B2 (en) 2019-06-20 2023-04-04 Cilag Gmbh International Wide dynamic range using a monochrome image sensor for fluorescence imaging
US11937784B2 (en) 2019-06-20 2024-03-26 Cilag Gmbh International Fluorescence imaging in a light deficient environment
US11172810B2 (en) * 2019-06-20 2021-11-16 Cilag Gmbh International Speckle removal in a pulsed laser mapping imaging system
US11012599B2 (en) * 2019-06-20 2021-05-18 Ethicon Llc Hyperspectral imaging in a light deficient environment
US11700995B2 (en) 2019-06-20 2023-07-18 Cilag Gmbh International Speckle removal in a pulsed fluorescence imaging system
US20200397277A1 (en) 2019-06-20 2020-12-24 Ethicon Llc Videostroboscopy of vocal cords with a hyperspectral, fluorescence, and laser mapping imaging system
US11624830B2 (en) 2019-06-20 2023-04-11 Cilag Gmbh International Wide dynamic range using a monochrome image sensor for laser mapping imaging
US11141052B2 (en) 2019-06-20 2021-10-12 Cilag Gmbh International Image rotation in an endoscopic fluorescence imaging system
JP7358817B2 (ja) * 2019-07-24 2023-10-11 ソニーグループ株式会社 画像処理装置、撮像装置、および画像処理方法、並びにプログラム
US20210404951A1 (en) * 2019-08-01 2021-12-30 Viavi Solutions Inc. Sensor device
US11980350B2 (en) * 2019-08-05 2024-05-14 Gyrus Acmi, Inc. Laser system with illumination control
EP3779554B1 (fr) * 2019-08-14 2024-01-17 Leica Instruments (Singapore) Pte. Ltd. Ensemble séparateur de faisceau optique, tête de caméra et ensemble microscope
US20210052161A1 (en) * 2019-08-23 2021-02-25 Rebound Therapeutics Corporation Fluorescence imaging system
DE102019123053A1 (de) * 2019-08-28 2021-03-04 Olympus Winter & Ibe Gmbh Endoskop mit optischer Filteranordnung und Verwendung
US11808703B2 (en) * 2019-10-03 2023-11-07 The Regents Of The University Of California Fluorescence imitating brightfield imaging
KR102351249B1 (ko) * 2020-01-03 2022-01-14 한국과학기술원 분자 다중 이미징 방법 및 장치
US20210275039A1 (en) * 2020-03-04 2021-09-09 Cardiac Pacemakers, Inc. Body vibration analysis systems and methods
EP3882681A1 (fr) * 2020-03-10 2021-09-22 Leica Instruments (Singapore) Pte. Ltd. Concept pour un système de microscope avec un système d'éclairage à base de del
KR102404070B1 (ko) * 2020-09-25 2022-06-02 고려대학교 산학협력단 광섬유 번들을 이용하는 반사 내시현미경 및 이를 이용한 이미지 획득 방법
DE102021106836A1 (de) * 2020-10-02 2022-04-07 Karl Storz Se & Co. Kg Optisches Filtersystem für ein Video-Endoskop, Anzeigesystem und Video-Endoskop
DE102020131374B4 (de) 2020-11-26 2024-02-08 Dionex Softron Gmbh Fluoreszenzdetektion
DE102020132978A1 (de) 2020-12-10 2022-06-15 Karl Storz Se & Co. Kg Erfassung von Bildern eines medizinischen Objekts in Weißlicht und Fluoreszenzlicht
DE102020132982A1 (de) 2020-12-10 2022-06-15 Karl Storz Se & Co. Kg Erfassung von Bildern eines medizinischen Objekts in Weißlicht und Fluoreszenzlicht
DE102020132951A1 (de) 2020-12-10 2022-06-15 Karl Storz Se & Co. Kg Erfassung von Bildern eines medizinischen Situs in Weißlicht und Fluoreszenzlicht
JPWO2022131224A1 (fr) * 2020-12-14 2022-06-23
CN113030038A (zh) * 2021-02-23 2021-06-25 北京理工大学 多波长荧光检测装置及方法
CN113456067B (zh) * 2021-06-24 2023-06-23 江西科莱富健康科技有限公司 体外血红蛋白浓度测量方法、系统及计算机设备
KR102624004B1 (ko) * 2021-11-16 2024-01-11 한국광기술원 형광영상 및 컬러영상 획득을 위한 다중 형광영상장치 및 그 제어방법
US20230351551A1 (en) * 2022-04-29 2023-11-02 Wisconsin Alumni Research Foundation Low-Light Video System
WO2024011209A2 (fr) * 2022-07-08 2024-01-11 Blaze Bioscience, Inc. Obturateur à cristaux liquides modifié utilisé comme filtre optique passe-long dynamique
WO2024123694A1 (fr) * 2022-12-09 2024-06-13 Stryker Corporation Systeme de visualisation par fluorescence visible
CN116725467B (zh) * 2023-03-31 2024-03-01 苏州宇懋医学科技有限公司 内窥镜装置、内窥镜医疗辅助系统及内窥镜图像处理方法

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002287034A (ja) * 2001-03-22 2002-10-03 Olympus Optical Co Ltd 走査型光学顕微鏡

Family Cites Families (49)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3659918A (en) 1970-03-24 1972-05-02 Philips Corp Color separating prism system
JPH07163572A (ja) * 1993-12-16 1995-06-27 Matsushita Electric Ind Co Ltd 光化学診断装置
DE19612536A1 (de) * 1996-03-29 1997-10-02 Freitag Lutz Dr Anordnung und Verfahren zur Diagnose von malignem Gewebe durch Fluoreszenzbetrachtung
JP2002500907A (ja) * 1998-01-26 2002-01-15 マサチユセツツ・インスチチユート・オブ・テクノロジイ 蛍光画像化用内視鏡
US6110106A (en) * 1998-06-24 2000-08-29 Biomax Technologies, Inc. Endoscopes and methods relating to direct viewing of a target tissue
US6727521B2 (en) 2000-09-25 2004-04-27 Foveon, Inc. Vertical color filter detector group and array
US7904139B2 (en) * 1999-08-26 2011-03-08 Non-Invasive Technology Inc. Optical examination of biological tissue using non-contact irradiation and detection
US6638668B2 (en) 2000-05-12 2003-10-28 Ocean Optics, Inc. Method for making monolithic patterned dichroic filter detector arrays for spectroscopic imaging
JP3695340B2 (ja) * 2001-03-30 2005-09-14 株式会社日立製作所 Ｄｎａ検査方法及びその装置並びに蛍光検出方法
US7352892B2 (en) * 2003-03-20 2008-04-01 Micron Technology, Inc. System and method for shape reconstruction from optical images
US20040225222A1 (en) * 2003-05-08 2004-11-11 Haishan Zeng Real-time contemporaneous multimodal imaging and spectroscopy uses thereof
EP1859239B1 (fr) * 2005-02-08 2017-09-06 Northrop Grumman Systems Corporation Systeme et procedes utilises dans la detection de particules aerosols nocives
FR2882147B1 (fr) * 2005-02-14 2007-10-05 Commissariat Energie Atomique Dispositif d'imagerie de fluorescence par reflexion a deux longueurs d'onde
JP2006263044A (ja) * 2005-03-23 2006-10-05 Fuji Photo Film Co Ltd 蛍光検出システム
EP2485052B1 (fr) 2005-09-13 2015-05-06 Affymetrix, Inc. Microparticules codées
US20130191368A1 (en) 2005-10-26 2013-07-25 c/o Cortica, Ltd. System and method for using multimedia content as search queries
EP1952106B1 (fr) 2005-11-04 2014-08-27 George Themelis Systeme d'imagerie multispectrale
JP4679375B2 (ja) * 2006-01-25 2011-04-27 株式会社日立ハイテクノロジーズ キャピラリ電気泳動装置
US20080062429A1 (en) * 2006-09-12 2008-03-13 Rongguang Liang Low coherence dental oct imaging
US8447087B2 (en) * 2006-09-12 2013-05-21 Carestream Health, Inc. Apparatus and method for caries detection
US8270689B2 (en) 2006-09-12 2012-09-18 Carestream Health, Inc. Apparatus for caries detection
US20080104352A1 (en) 2006-10-31 2008-05-01 Advanced Micro Devices, Inc. Memory system including a high-speed serial buffer
DE102006056429B3 (de) * 2006-11-28 2008-02-14 Leica Microsystems Cms Gmbh Lasermikroskop mit räumlich trennendem Strahlteiler
US8498695B2 (en) 2006-12-22 2013-07-30 Novadaq Technologies Inc. Imaging system with a single color image sensor for simultaneous fluorescence and color video endoscopy
JP4954699B2 (ja) * 2006-12-28 2012-06-20 オリンパス株式会社 蛍光内視鏡システム
WO2009005748A1 (fr) * 2007-06-29 2009-01-08 The Trustees Of Columbia University In The City Ofnew York Systèmes et procédés d'imagerie optique ou de spectroscopie
JP4993195B2 (ja) 2007-07-02 2012-08-08 株式会社イノアックコーポレーション ブロー成形品の穿孔方法とブロー成形用金型
US8098375B2 (en) * 2007-08-06 2012-01-17 Lumencor, Inc. Light emitting diode illumination system
JP2009122203A (ja) * 2007-11-12 2009-06-04 Shimadzu Corp 光学フィルタ、蛍光分析用フィルタセット及びその光学フィルタを用いた蛍光測定装置
US8849380B2 (en) * 2007-11-26 2014-09-30 Canfield Scientific Inc. Multi-spectral tissue imaging
WO2013109966A1 (fr) * 2012-01-20 2013-07-25 The Trustees Of Dartmouth College Procédé et appareil pour imagerie quantitative hyperspectrale par fluorescence et réflectance pour guidage chirurgical
WO2010019515A2 (fr) * 2008-08-10 2010-02-18 Board Of Regents, The University Of Texas System Appareil d'imagerie hyperspectrale à traitement de lumière numérique
JP5342869B2 (ja) * 2008-12-22 2013-11-13 Ｈｏｙａ株式会社 内視鏡装置、内視鏡照明装置、画像形成装置、内視鏡照明装置の作動方法および画像形成装置の作動方法
JP2010273301A (ja) * 2009-05-25 2010-12-02 Pfu Ltd 画像読取装置
US9749607B2 (en) * 2009-07-16 2017-08-29 Digimarc Corporation Coordinated illumination and image signal capture for enhanced signal detection
JPWO2011048886A1 (ja) * 2009-10-20 2013-03-07 オリンパスメディカルシステムズ株式会社 蛍光観察装置
JP5562683B2 (ja) 2010-03-03 2014-07-30 オリンパス株式会社 蛍光観察装置
US9654745B2 (en) * 2010-03-17 2017-05-16 Haishan Zeng Rapid multi-spectral imaging methods and apparatus and applications for cancer detection and localization
JP5538072B2 (ja) * 2010-06-03 2014-07-02 株式会社日立ハイテクノロジーズ 欠陥検査方法およびその装置
JP5707758B2 (ja) 2010-07-13 2015-04-30 ソニー株式会社 撮像装置、撮像システム、手術用ナビゲーションシステム、及び撮像方法
DE102010060121C5 (de) * 2010-10-22 2023-09-28 Leica Microsystems Cms Gmbh SPIM-Mikroskop mit sequenziellem Lightsheet
JP5274720B2 (ja) 2010-12-14 2013-08-28 オリンパスメディカルシステムズ株式会社 撮像装置
US20150087902A1 (en) * 2012-03-30 2015-03-26 Trustees Of Boston University Phase Contrast Microscopy With Oblique Back-Illumination
US9593982B2 (en) * 2012-05-21 2017-03-14 Digimarc Corporation Sensor-synchronized spectrally-structured-light imaging
JP5996287B2 (ja) * 2012-06-12 2016-09-21 オリンパス株式会社 撮像装置、顕微鏡装置、内視鏡装置
US10376147B2 (en) * 2012-12-05 2019-08-13 AccuVeiw, Inc. System and method for multi-color laser imaging and ablation of cancer cells using fluorescence
US9068949B2 (en) * 2013-02-04 2015-06-30 Purdue Research Foundation System and method for multiplex spectroscopic imaging
US9155474B2 (en) * 2013-08-14 2015-10-13 Omega Optical, Inc. System for multispectral imaging of fluorescence
CA3009419A1 (fr) * 2016-01-26 2017-08-03 Novadaq Technologies ULC Plateforme configurable

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002287034A (ja) * 2001-03-22 2002-10-03 Olympus Optical Co Ltd 走査型光学顕微鏡

Also Published As

Publication number Publication date
EP3152549B1 (fr) 2023-06-07
JP6728070B2 (ja) 2020-07-22
JP2017529514A (ja) 2017-10-05
EP3152549C0 (fr) 2023-06-07
US10481095B2 (en) 2019-11-19
FI3940371T3 (fi) 2023-11-20
US20170176336A1 (en) 2017-06-22
CA2951075A1 (fr) 2015-12-10
ES2964016T3 (es) 2024-04-03
EP3940371A1 (fr) 2022-01-19
CA2951086A1 (fr) 2015-12-10
ES2883220T3 (es) 2021-12-07
EP3940371B1 (fr) 2023-08-30
JP2017526899A (ja) 2017-09-14
US10684224B2 (en) 2020-06-16
WO2015185662A2 (fr) 2015-12-10
EP3152548A1 (fr) 2017-04-12
CN106999021A (zh) 2017-08-01
WO2015185661A1 (fr) 2015-12-10
CN106572792A (zh) 2017-04-19
CN106572792B (zh) 2020-03-06
EP3152548B1 (fr) 2021-05-19
CN106999021B (zh) 2021-06-04
WO2015185662A3 (fr) 2016-07-21
JP6696912B2 (ja) 2020-05-20
DK3940371T3 (da) 2023-11-27
US20170167980A1 (en) 2017-06-15

Similar Documents

Publication Publication Date Title
EP3152549B1 (fr) Procédé et moyen d'imagerie multispectrale
US11206987B2 (en) Method and apparatus for concurrent imaging at visible and infrared wavelengths
US20160262602A1 (en) Laparoscopic Cholecystectomy With Fluorescence Cholangiography
US8382812B2 (en) Apparatus for photodynamic therapy and photodetection
EP2108943B1 (fr) Dispositif et procédé d'imagerie à fluorescence
JP5229723B2 (ja) 蛍光画像取得装置
JP4818753B2 (ja) 内視鏡システム
US20120248333A1 (en) Device For Fluorescence Diagnosis
US20110270092A1 (en) Combined apparatus for detection of multispectral optical image emitted from living body and for light therapy
US20080177140A1 (en) Cameras for fluorescence and reflectance imaging
JP2014087661A (ja) 画像化内視鏡と共に使用するフィルター
IL173288A (en) A device for detecting and characterizing biological tissue
JP2007313171A (ja) 内視鏡システム
JPWO2009072177A1 (ja) 分光観察装置、内視鏡システムおよびカプセル内視鏡システム
JP2022527642A (ja) 狭帯域光観察を利用する医療機器

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170104

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RIN1 Information on inventor provided before grant (corrected)

Inventor name: DIMITRIADIS, NIKOLAS

Inventor name: DELIOLANIS, NIKOLAOS

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190625

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20220727

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTC Intention to grant announced (deleted)
INTG Intention to grant announced

Effective date: 20221102

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1576529

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230615

Ref country code: DE

Ref legal event code: R096

Ref document number: 602015083858

Country of ref document: DE

U01 Request for unitary effect filed

Effective date: 20230628

U07 Unitary effect registered

Designated state(s): AT BE BG DE DK EE FI FR IT LT LU LV MT NL PT SE SI

Effective date: 20230706

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230907

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231007

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231007

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602015083858

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20240308