EP3859656B1 - System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops - Google Patents

System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops Download PDF

Info

Publication number
EP3859656B1
EP3859656B1 EP21154120.6A EP21154120A EP3859656B1 EP 3859656 B1 EP3859656 B1 EP 3859656B1 EP 21154120 A EP21154120 A EP 21154120A EP 3859656 B1 EP3859656 B1 EP 3859656B1
Authority
EP
European Patent Office
Prior art keywords
image data
raw image
pixel
intensity
optical imaging
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP21154120.6A
Other languages
English (en)
French (fr)
Other versions
EP3859656A1 (de
Inventor
Stanislaw Warych
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Leica Instruments Singapore Pte Ltd
Original Assignee
Leica Instruments Singapore Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Leica Instruments Singapore Pte Ltd filed Critical Leica Instruments Singapore Pte Ltd
Priority to EP25156597.4A priority Critical patent/EP4576804A3/de
Publication of EP3859656A1 publication Critical patent/EP3859656A1/de
Application granted granted Critical
Publication of EP3859656B1 publication Critical patent/EP3859656B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H04N25/67—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4015—Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10064—Fluorescence image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing
    • G06T2207/20012—Locally adaptive
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20024—Filtering details
    • G06T2207/20032—Median filtering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing

Definitions

  • Examples relate to a system, a method and a computer program for processing raw image data of a microscope, and to a corresponding microscope system
  • Signal noise decreases the value of the information obtained from a sensor, such as an optical imaging sensor, which is available for the user.
  • signal noise can occur due to multiple reasons, such as random electron motion due to heat, or due to sensor dust.
  • Denoising refers to a method for addressing discovered sources of the noise in the signal to improve an overall signal to noise ratio. Signal denoising is a very wide field by itself and generic approaches are known that deal with signal noise.
  • US patent application US 2011/0187902 A1 relates to a concept for denoising color filter array (CFA) images using weighted pixel differences.
  • CFA color filter array
  • denoising is applied before CFA reconstruction.
  • Kalevo et al. "Noise Reduction Techniques for Bayer-Matrix Images"
  • three image processing pipelines are compared, one where denoising is performed before CFA reconstruction, and two where denoising is performed after CFA reconstruction, with the denoising being performed using linear, multi-stage median, multistage median hybrid and median-rational filters.
  • Bosco et al "Noise Reduction for CFA Image Sensors Exploiting HVS Behavior”
  • denoising is also performed before CFA reconstruction.
  • Microscopes as an optical system, have its own specific sources of noise. Other approaches might not account for denoising images originating from microscopes.
  • Richard Bowman et al. "Flat-field and colour correction for the Raspberry Pi camera module" introduces various techniques for improving the image quality of the Raspberry Pi camera module when the camera module is used with other optics, e.g., to perform microscopy. It references the image processing pipeline used by the Raspberry Pi camera module, which includes two denoising steps, one before demosaicing and another after demosaicing.
  • the optical imaging sensor may have sensor pixels that sense light in different wavelength bands. Based on the wavelength band the respective sensor pixels sense, the sensor pixels may be assigned to a group of sensor pixels, with each group comprising sensor pixels that sense light in a specific wavelength band. Light in that wavelength band that is incident to other sensor pixels (not part of the group) may be sensed by interpolating the sensor values of sensor pixels of the group that are located adjacent to the respective sensor pixel. This is done in a process that is denoted "filter pattern reconstruction", which refers to the pattern of the filter that is being used to make the respective sensor pixels sense light in the different wavelength bands.
  • filter pattern reconstruction refers to the pattern of the filter that is being used to make the respective sensor pixels sense light in the different wavelength bands.
  • Embodiments are based on the finding, that, in order to improve the de-noising of image data of a microscope, the de-noising is performed before filter pattern reconstruction, in order to limit the influence of noise to a specific pixel, instead of spreading the noise to multiple pixels during the reconstruction.
  • Embodiments of the present disclosure provide a system for processing raw image data of a microscope.
  • the system comprises one or more processors and one or more storage devices.
  • the system is configured to obtain the raw image data from an optical imaging sensor of the microscope.
  • the raw image data is based on a filter pattern of the optical imaging sensor.
  • the system is configured to perform de-noising of the raw image data to obtain de-noised raw image data.
  • the system is configured to perform filter pattern reconstruction using the de-noised raw image data to obtain reconstructed image data. By performing the filter pattern reconstruction based on the de-noised data, a spreading of the noise to multiple pixels may be avoided.
  • the de-noising of the raw image data is performed before the filter pattern reconstruction.
  • a spreading of the noise to multiple pixels may be avoided.
  • the raw image data is based on a plurality of pixels.
  • the plurality of pixels may represent light in two or more wavelength bands, such that the raw image data comprises, for each pixel, an intensity value representing an intensity of light in one of the two or more wavelength bands.
  • the de-noising may be performed separately in at least one of the two or more wavelength bands, e.g. for each of the two or more wavelength bands, based on the intensity values of the pixels of the respective wavelength band. This way, de-noising may be performed separately, or exclusively, for wavelength bands being used for fluorescence imaging, without accounting for noise of the other wavelength bands.
  • performing the de-noising comprises applying a median filter to de-noise the raw image data.
  • a median filter may be used to reduce the influence of outliers (and thus noise) in the raw image data.
  • the raw image data is based on a plurality of pixels.
  • the plurality of pixels may represent light in two or more wavelength bands, such that the raw image data comprises, for each pixel, an intensity value representing an intensity of light in one of the two or more wavelength bands.
  • Applying the median filter may comprise, for each pixel of the plurality of pixels, calculating a median value of the intensity of the light represented by the pixel and the intensity of light of one or more adjacent pixels representing light in the same wavelength band as the pixel, and using the median value for a corresponding pixel of the de-noised raw image data. This may implement a filter pattern-specific denoising of the raw image data using a median filter.
  • performing the de-noising may comprise applying an averaging filter to de-noise the raw image data.
  • Averaging may be used in some embodiments instead of a median filter.
  • the raw image data is based on a plurality of pixels.
  • the raw image data comprises for each pixel, an intensity value representing an intensity of light.
  • Performing the de-noising comprises comparing the intensity value of a pixel of the plurality of pixels to a pixel-specific intensity threshold, and adapting the intensity value of the pixel if the intensity is at most the pixel-specific threshold. This may suppress background noise that is due to the construction of the microscope.
  • the intensity value may be compared after applying an averaging or median filter to de-noise the raw image data. In this order, an effectiveness of the de-noising may be improved.
  • the intensity value of the pixel may be discarded if the intensity is at most the pixel-specific threshold. This may enable the use of a pixel-specific background value for background noise suppression.
  • the pixel-specific threshold is based on an optical imaging sensor-specific pattern of noise.
  • the de-noising may be used to suppress background noise that is due to the construction of the microscope, based on a pattern that is specific at least to the type of microscope.
  • the optical imaging sensor-specific pattern of noise may be a generic pattern that is suitable for optical imaging sensors of different microscopes of the same type of microscope.
  • a suitable pattern may be provided without requiring measurements within the specific microscope it is being used in.
  • the optical imaging sensor-specific pattern of noise may be a pattern that is generated using the optical imaging sensor of the microscope. Such a pattern may be more effective for the specific microscope but may require additional measurements.
  • the optical imaging sensor-specific pattern of noise may be a pattern that is generated during factory calibration of the optical imaging sensor.
  • the system may be configured to generate the optical imaging sensor-specific pattern of noise using the optical imaging sensor.
  • the optical imaging sensor-specific pattern of noise may be generated by the system.
  • the de-noising is performed for raw image data of fluorescence images.
  • Fluorescence images may be susceptible to noise, as the light intensity is typically lower than in reflectance images.
  • the filter pattern is a Bayer pattern.
  • the system may be configured to perform Bayer reconstruction using the de-noised raw image data.
  • Bayer patterns are often used in optical imaging sensors of microscopes.
  • Embodiments of the present disclosure further provide a microscope system comprising the above system and a microscope.
  • Embodiments of the present disclosure further provide a method for processing raw image data of a microscope.
  • the method comprises obtaining the raw image data from an optical imaging sensor of the microscope.
  • the raw image data is based on a filter pattern of the optical imaging sensor.
  • the method comprises performing de-noising of the raw image data to obtain de-noised raw image data.
  • the method comprises performing filter pattern reconstruction using the de-noised raw image data to obtain reconstructed image data.
  • Embodiments of the present disclosure provide a computer program with a program code for performing the method when the computer program is executed on a processor.
  • Fig. 1a shows a block diagram of a system 110 for processing raw image data of a microscope.
  • the system comprises one or more processors 114 and one or more storage devices 116.
  • the system may further comprise an interface 112.
  • the one or more processors 114 are coupled to the one or more storage devices and to the interface 112.
  • the one or more processors 114 are configured to perform the respective actions of the system, e.g. in conjunction with the one or more storage devices 116 and/or the interface 112.
  • the system is configured to obtain the raw image data from an optical imaging sensor 122 of the microscope.
  • the raw image data is based on a filter pattern of the optical imaging sensor.
  • the system is configured to perform de-noising of the raw image data to obtain de-noised raw image data.
  • the system is configured to perform filter pattern reconstruction using the de-noised raw image data to obtain reconstructed image data.
  • Fig. 1b shows a block diagram of a microscope system comprising the microscope 120 and the system 110 for processing raw image data of a microscope.
  • the microscope system shown in Fig. 1b is a surgical microscope system, which may be used at a surgical site by a surgeon.
  • the surgical microscope system shown in Fig. 1b comprises a number of optional components, such as a base unit 105 (comprising the system 110) with a (rolling) stand, an auxiliary display 130, a (robotic or manual) arm 140 which holds the microscope 120 in place, and which is coupled to the base unit 105 and to the microscope 120, and steering handles 150 that are attached to the microscope 120.
  • the microscope 120 may comprise optional ocular displays 124 and an optional auxiliary display 126.
  • the term "(surgical) microscope system” is used, in order to cover the portions of the system that are not part of the actual microscope (which comprises optical components), but which are used in conjunction with the microscope, such as the display or a lighting system.
  • Embodiments of the present disclosure relate to a system, method and computer program for processing raw image data of a microscope 120.
  • a microscope is an optical instrument that is suitable for examining objects that are too small to be examined by the human eye (alone).
  • a microscope may provide an optical magnification of an object.
  • the optical magnification is often provided for a camera or an imaging sensor, such as the optical imaging sensor of the microscope 120 of Fig. 1a .
  • the microscope 120 may further comprise one or more optical magnification components that are used to magnify a view on the sample.
  • the object being viewed through the microscope may be a sample of organic tissue, e.g. arranged within a petri dish or present in a part of a body of a patient.
  • the microscope system 100 may be a microscope system for use in a laboratory, e.g. a microscope that may be used to examine the sample of organic tissue in a petri dish.
  • the respective sensor pixel might only sense light in the wavelength band admitted by the respective optical filter, and an output of the sensor pixel may represent an intensity of the light sensed by the sensor pixel.
  • the two or more wavelength bands may comprise a "red" wavelength band (i.e. a portion of the wavelength spectrum comprising red light), a "green” wavelength band (i.e. a portion of the wavelength spectrum comprising green light) and a "blue” wavelength band (i.e. a portion of the wavelength spectrum comprising blue light).
  • the raw image data comprises, for each pixel, an intensity value representing an intensity of light in (only/exactly) one of the red, green and blue wavelength band.
  • the raw image data may be image data that contains, for each pixel of the image data, only an intensity value for a single wavelength band (and not for three wavelength bands, such as Red/Green/Blue).
  • the system is configured to perform de-noising of the raw image data to obtain de-noised raw image data.
  • De-noising or noise reduction, refers to a process suitable for reducing an amount of sensor noise in sensor data, e.g. in the image data provided by the optical imaging system.
  • the noise often originates from random electron motion due to thermal agitation (i.e. in heated optical imaging sensors), from dust or from light or heat leakage within the optical system (i.e. the microscope). It often manifests itself as Salt-and-Pepper noise (noise that produces outliers in an otherwise homogenic environment) or as Gaussian noise (noise that effects a larger number of sensor pixels by a small amount).
  • de-noising may be performed for salt-and-pepper type noise (i.e. to eliminate outliers affecting individual sensor pixels in the raw image data). In general, one or multiple de-noising actions may be taken.
  • the de-noising may be performed separately in at least one of the two or more wavelength bands, e.g. for each of the two or more wavelength bands, based on the intensity values of the pixels of the respective wavelength band.
  • the at least one of the two or more wavelength bands may be tuned to, e.g. coincide with, a fluorescence emission wavelength band of a fluorophore. This way, the de-noising may be performed (only and) separately for the wavelength band or bands being used for fluorescence imaging.
  • the de-noising may also be performed based on the intensity values alone, without reference to a color they represent.
  • the de-noising may be performed (directly) on the intensity values of the pixels.
  • the de-noising may be performed without mapping the intensity values into a color space, or without converting between different color spaces.
  • a median filter or an averaging filter may be used for de-noising.
  • performing the de-noising may comprise applying a median filter to de-noise the raw image data.
  • performing the de-noising comprises applying an averaging filter to de-noise the raw image data.
  • the course of action is similar - for each pixel, a plurality of adjacent pixels that sense light in the same wavelength band are evaluated, and either a median or an average value (for the intensity of light) is calculated based on the values of the pixel and of the adjacent pixels.
  • the raw image data may comprise, for each pixel, an intensity value representing an intensity of light in (exactly) one of the two or more wavelength bands.
  • each pixel has four adjacent pixels sensing light in the same wavelength band (at least for pixels removed from the edges of the optical imaging sensors), which are located once removed in every dimension in a diagonal manner for red and green, and twice removed in a non-diagonal manner for blue.
  • "adjacent" pixels need not be directly adjacent to the pixel the de-noising is performed on, but can be removed by at least one further pixel, or be adjacent in a diagonal manner.
  • adjacent pixels might not be removed by more than e.g. three pixels in any dimension.
  • the de-noising of the raw image data may comprise iterating over (all of) the plurality of pixels to (individually) de-noise each pixel of the plurality of pixels. Close to the edges of the optical imaging sensor, fewer adjacent pixels might be used.
  • the filter pattern may be a Bayer pattern, or another pattern, such as a RGEB (Red, Green, Emerald, Blue) pattern, or a modified Bayer pattern having different sizes for the filters, such as an XTrans filter pattern used by Fujifilm.
  • RGEB Red, Green, Emerald, Blue
  • additional de-noising actions might be taken.
  • de-noising may be performed that addresses known sources of noise.
  • background noise may be present in the raw image data that is due to the intrinsic properties (i.e. the construction) of the optical imaging system.
  • Such noise may be removed by generating an optical imaging sensor-specific pattern of noise, and using that pattern to suppress light intensity values that are likely caused by the background noise.
  • the raw image data may be based on a plurality of pixels.
  • the raw image data may comprise, for each pixel, an intensity value representing an intensity of light (of light in any wavelength band).
  • Performing the de-noising may comprise comparing the intensity value of a pixel of the plurality of pixels to a pixel-specific intensity threshold, and adapting (e.g. suppressing, adjusting or discarding) the intensity value of the pixel if the intensity is at most the pixel-specific threshold.
  • the intensity value may be compared after (or alternatively before) applying an averaging or median filter to de-noise the raw image data. For example, the intensity value of the pixel may be discarded/suppressed if the intensity is at most the pixel-specific threshold. In other words, the intensity value of a pixel may be adjusted if the pixel-specific intensity threshold indicates that the pixel has (primarily or only) sensed background noise.
  • the term pixel-specific intensity threshold may be understood as an intensity threshold that is defined for the pixel.
  • the intensity threshold may be defined for groups of pixels (e.g. a group of six pixels of a cell of the Bayer pattern, or a group of 2x2 pixels etc.). In some embodiments, however, the intensity threshold may be defined individually for every single pixel of the plurality of pixels.
  • an intensity threshold may be defined by an optical imaging sensor-specific threshold map/array, which may be based on the background noise of the optical imaging sensor (within the microscope).
  • the pixel-specific threshold may be based on an optical imaging sensor-specific pattern of noise.
  • the optical imaging sensor-specific pattern of noise may represent, or be based on, a background noise that the optical imaging sensor perceives during its operation, e.g. once the system has warmed up.
  • the optical imaging sensor-specific pattern of noise may comprise an array/a matrix of intensity threshold values for the plurality of pixels. This array/matrix of intensity threshold values may represent, or be based on, the background noise that the optical imaging sensor perceives during its operation.
  • optical imaging sensor-specific patterns of noise may be used for de-noising the raw image data.
  • an optical imaging sensor-specific pattern of noise might be used that is suitable/applicable for the type of microscope, i.e. that can be used for different microscopes of the same type.
  • the optical imaging sensor-specific pattern of noise may be a generic pattern that is suitable for optical imaging sensors of different microscopes of the same type of microscope.
  • the term "generic pattern" indicates, that the optical imaging sensor-specific pattern of noise is suitable for more than one optical imaging sensor.
  • the optical imaging sensor-specific pattern of noise might not be specific to the very microscope it is being used it, but to the type of microscope of the microscope it is being used in.
  • optical imaging sensor-specific pattern of noise may be used that is specific to the very microscope and optical imaging sensor it is being used in.
  • the optical imaging sensor-specific pattern of noise may be generated using the optical imaging sensor of the microscope.
  • the optical imaging sensor-specific pattern of noise may be a pattern that is generated using the optical imaging sensor of the microscope.
  • the optical imaging sensor-specific pattern of noise may be specific to, or calibrated for, the optical imaging sensor of the microscope.
  • the optical imaging sensor-specific pattern of noise might be suitable for/tailored to a single microscope/optical imaging sensor.
  • the optical imaging sensor-specific pattern of noise may be a pattern that is generated during factory calibration of the optical imaging sensor.
  • the optical imaging sensor-specific pattern of noise may be generated by a calibration apparatus that is external to the microscope system 100 (e.g. a calibration apparatus that is part of the assembly line for manufacturing the microscope system), or by the system 110. Accordingly, the system is configured to generate the optical imaging sensor-specific pattern of noise using the optical imaging sensor, e.g. during factory calibration, or during a later calibration. During the generation of the optical imaging sensor-specific pattern of noise, the illumination sources of the microscope system may be disabled, and ambient light might be avoided. Additionally, the system may be in operation for some time (e.g. an hour) before the generation of the optical imaging sensor-specific pattern of noise, to make sure heat has spread through the microscope in a realistic manner. Once the microscope has reached realistic heat distribution, the optical imaging sensor-specific pattern of noise may be generated from raw image data of the optical imaging sensor, with the optical imaging sensor recording a black background in darkness.
  • the de-noising, or the comparison with the pixel-specific intensity values might be performed (only) for raw image data of fluorescence images (i.e. images wherein the sensed lights is due to fluorescence of the object being recorded).
  • the de-noising, or the comparison with the pixel-specific intensity values might be omitted for raw image data of reflectance images (i.e. images wherein the sensed light is light that is reflected by the object being recorded).
  • the system is configured to perform filter pattern reconstruction using the de-noised raw image data to obtain reconstructed image data.
  • the de-noising of the raw image data is performed before the filter pattern reconstruction.
  • the filter pattern reconstruction may be performed according to the filter pattern that was used to record the raw image data. For example, if the filter pattern is a Bayer pattern, the system may be configured to perform Bayer reconstruction using the de-noised raw image data. With other filter patterns, the respective reconstruction algorithms may be used. An example is given for Bayer pattern reconstruction with reference to Fig. 3 .
  • the interface 112 may correspond to one or more inputs and/or outputs for receiving and/or transmitting information, which may be in digital (bit) values according to a specified code, within a module, between modules or between modules of different entities.
  • the interface 112 may comprise interface circuitry configured to receive and/or transmit information.
  • the one or more processors 114 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software.
  • the described function of the one or more processors 114 may as well be implemented in software, which is then executed on one or more programmable hardware components.
  • Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, etc.
  • the one or more storage devices 116 may comprise at least one element of the group of a computer readable storage medium, such as an magnetic or optical storage medium, e.g. a hard disk drive, a flash memory, Floppy-Disk, Random Access Memory (RAM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), an Electronically Erasable Programmable Read Only Memory (EEPROM), or a network storage.
  • a computer readable storage medium such as an magnetic or optical storage medium, such as an magnetic or optical storage medium, e.g. a hard disk drive, a flash memory, Floppy-Disk, Random Access Memory (RAM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), an Electronically Erasable Programmable Read Only Memory (EEPROM), or a network storage.
  • the system or microscope system may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.
  • Fig. 2 shows a flow chart of a (corresponding) method for processing raw image data of a microscope.
  • the microscope may be implemented similar to the microscope of Fig. 1b .
  • the method comprises obtaining 210 the raw image data from an optical imaging sensor of the microscope.
  • the raw image data is based on a filter pattern of the optical imaging sensor.
  • the method comprises performing 220 de-noising of the raw image data to obtain de-noised raw image data.
  • the method comprises performing 250 filter pattern reconstruction using the de-noised raw image data to obtain reconstructed image data.
  • the method may optionally comprise calculating 230 a median or average value of the intensity of the light represented by the pixel and the intensity of light of one or more adjacent pixels representing light in the same wavelength band as the pixel, and using 235 the median/average value for the corresponding pixel of the de-noised raw image data.
  • the method may comprise comparing 240 the intensity value of a pixel of the plurality of pixels to a pixel-specific intensity threshold, and adapting 245 the intensity value of the pixel if the intensity is at most the pixel-specific threshold.
  • the method may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.
  • Embodiments of the present disclosure provide a system, method and computer program for improving a noise to signal ratio, i.e. for image denoising.
  • Embodiments may be based on applying two step denoising on image obtained by the camera.
  • the first step may compensate for a source of the noise in image acquisition by using a median filter.
  • the median filter may be applied before image reconstruction is being done by the camera, directly on the RAW/filter pattern (e.g. the Bayer pattern).
  • the optional second step addresses a specific source of the noise, originating from the optics inside the microscope.
  • a median filter may be applied for Bayer pattern images.
  • the first step in the noise reduction uses a median filter.
  • a median filter is typically used to eliminate salt and pepper type of noise, which can originate for example from analog to digital conversion or current leakage in digital cameras.
  • the majority of digital cameras use a filter pattern, such as a Bayer pattern, in front of the sensor to reduce cost and size of the device with minimal impact to image quality.
  • a Bayer pattern camera has one sensor hence three times less information than a three-sensor camera solution (dedicated sensors for red, green and blue wavelengths), but with special design of the pattern and dedicated reconstruction algorithms it can perform close to solutions with three dedicated sensors.
  • Common path to achieve denoised image in case of camera with Bayer pattern is as follows: Obtain Filter Pattern (e.g. Bayer) image (e.g. the raw image data) ⁇ Apply reconstruction algorithm ⁇ Use median filter with RGB image.
  • FIG. 3 shows an illustrative diagram of a (Bayer) filter pattern.
  • pixels 1, 3, 5, 11, 13, 15, 21, 23 and 25 sense light in a first wavelength band (e.g. red light), even-numbered pixels sense light in a second wavelength band (e.g. green light) and pixels 7, 9, 17 and 19 sense light in a third wavelength band (e.g. blue light).
  • Embodiments may overcome this possible drawback by changing the chain to: Obtain Filter Pattern (e.g. Bayer) image (e.g. the raw image data) ⁇ Use filter pattern specific (e.g. Bayer pattern specific) median filter (to de-noise the raw image data) ⁇ Apply reconstruction algorithm
  • Obtain Filter Pattern e.g. Bayer
  • Use filter pattern specific e.g. Bayer pattern specific
  • median filter to de-noise the raw image data
  • the pattern specific median filter may be applied first for (all of) the red input pixels (3, 5, 13 and 15).
  • the red pixel in the center it may use values of red pixels 1, 3, 5, 11, 13, 15, 21, 23, 25 to eliminate possible noise from 13 and this new value may be used as input to the reconstruction algorithm (same applies to red input pixels 3, 5 and 15 with respective neighborhood).
  • a filter pattern image e.g. Bayer pattern image
  • a filter pattern image that is denoised in such a manner may be used as input to the reconstruction algorithm.
  • the method comprises dark frame suppression as an optional second step.
  • the next step in the denoising scheme may target a source of the noise specific to the optics and fluorescence camera in the microscope.
  • signals that have a very high value for the user may be in the range of infra-red wavelength.
  • a dedicated camera for capturing the light in the infrared wavelength range may be part of the microscope. It may be desirable to avoid unwanted sources of infra-red wavelength inside the optics. Due to optics carrier construction and heat generated by the operating camera, it may be possible to observe "background" noise caused by internal heat. To compensate for it, measurements of dark frames may be taken. With switched off light sources and totally covered optics the images from the digital camera (i.e.
  • the optical imaging sensor may be captured. In ideal conditions they may be perfectly black. As they typically are not, this information may be input to extract a value which indicates a dark frame noise level (e.g. an optical imaging sensor-specific pattern of noise).
  • the calculated value e.g. the optical imaging sensor-specific pattern of noise
  • the intensity values of the respective pixel may be compared to the suppression value of the respective pixel, and the intensity value may be adapted (e.g. discarded / suppressed) if the intensity value is at most the suppression value.
  • the steps may be performed in a different order, e.g. the second step may be performed after the first step, or the two steps may be performed intertwined or in a combined manner, e.g. in a single operation.
  • a benefit of applying denoising before fluorescence extraction may be based on the fact that any noise that will fall through to fluorescence extraction can potentially introduce a false fluorescence signal or suppress an existing one.
  • the proposed approach it may be emphasized that only one specific color may be expected (and processed) in fluorescence imaging, and any deviation from RGB values encoding may be treated as noise whilst in white light (i.e. reflectance) imaging mode any deviation from RGB may be treated as a different, but still valid color.
  • Fig. 3b shows the sensor's spectral response, for blue pixels 310, green pixels 320 and red pixel 330, with depicted filtering out of wavelengths outside the signal component 340 comprising the fluorescence emissions. It can be seen that any signal that is expected to be used for fluorescence emissions in the wavelength bands follows some RGB curves properties. The requirement of fulfilling these properties is captured by the fluorescence extraction equation.
  • a constant measurement over the entire sensor area with one noisy pixel in the center (40) is assumed.
  • red pixels are represented by squares with an upward-facing line pattern
  • green pixels are represented by squares with a downward-facing line pattern
  • blue pixels are represented by squares with a dotted pattern.
  • the noisy pixel in the center (40) is a red pixel.
  • Fig. 3c shows the input pattern with noisy value (40) in the center (red pixel).
  • the resulting value for entire area, using specified equation, without noisy pixel should equal 310 (assuming a linear interpolation approach for the de-bayering step).
  • Fig. 3d shows the expected outcome.
  • the image may first be de-bayered, then the pixels may be extracted and a median filter may be applied. After de-bayering and extraction, the noisy pixel has an impact on neighborhood extraction, as shown in Fig. 3d .
  • a median (3x3) filter cannot restore the expected outcome, as shown in Fig. 3f . It may be possible to perform the restoration with two-step median filtering or with median filtering using a bigger neighborhood, but these approaches have a substantially higher computational complexity.
  • Figs. 3g and 3h the effect of the proposed modified approach is shown.
  • the median filter is applied separately for each channel, then de-bayering is performed, and then extraction.
  • Fig. 3g shows the color channel limited median filter output.
  • Fig. 3h shows the values after de-bayering and extraction.
  • the concept may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.
  • a microscope comprising a system as described in connection with one or more of the Figs. 1 to 3h .
  • a microscope may be part of or connected to a system as described in connection with one or more of the Figs. 1 to 3h .
  • Fig. 4 shows a schematic illustration of a (microscope) system 400 configured to perform a method described herein.
  • the system 400 comprises a microscope 410 (e.g. the microscope 120 of Figs. 1a and/or 1b) and a computer system 420 (e.g. the system 110 of Figs. 1a and/or 1b).
  • the microscope 410 is configured to take images and is connected to the computer system 420.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Optics & Photonics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Microscoopes, Condenser (AREA)
  • Image Processing (AREA)

Claims (14)

  1. System (110) zur Verarbeitung von Rohbilddaten eines Mikroskops (120), wobei das System einen oder mehrere Prozessoren (114) und eine oder mehrere Speichervorrichtungen (116) umfasst, wobei das System konfiguriert ist, um:
    die Rohbilddaten von einem optischen Abbildungssensor (122) des Mikroskops zu erhalten, wobei die Rohbilddaten auf einem Filtermuster des optischen Abbildungssensors basieren, wobei die Rohbilddaten auf einer Vielzahl von Pixeln basieren, wobei die Rohbilddaten für jedes Pixel einen Intensitätswert umfassen, der eine Lichtintensität darstellt;
    ein Entrauschen der Rohbilddaten durchzuführen, um entrauschte Rohbilddaten zu erhalten, wobei der Vorgang des Durchführens des Entrauschens das Vergleichen des Intensitätswerts eines Pixels der Vielzahl von Pixeln mit einem pixelspezifischen Intensitätsschwellenwert und das Anpassen des Intensitätswerts des Pixels umfasst, wenn die Intensität höchstens dem pixelspezifischen Schwellenwert entspricht; und
    Filtermusterrekonstruktion unter Verwendung der entrauschten Rohbilddaten durchzuführen, um rekonstruierte Bilddaten zu erhalten.
  2. System nach Anspruch 1, wobei das Entrauschen der Rohbilddaten vor der Rekonstruktion des Filtermusters durchgeführt wird.
  3. System nach einem der Ansprüche 1 oder 2, wobei die Rohbilddaten auf einer Vielzahl von Pixeln basieren, wobei die Vielzahl von Pixeln Licht in zwei oder mehr Wellenlängenbändern darstellt, so dass die Rohbilddaten für jedes Pixel einen Intensitätswert umfassen, der die Lichtintensität in einem der zwei oder mehr Wellenlängenbänder darstellt, wobei das Entrauschen getrennt für mindestens eines der zwei oder mehr Wellenlängenbänder auf der Grundlage der Intensitätswerte der Pixel des jeweiligen Wellenlängenbandes durchgeführt wird.
  4. System nach einem der Ansprüche 1 bis 3, wobei das Durchführen des Entrauschens das Anwenden eines Medianfilters zum Entrauschen der Rohbilddaten umfasst.
  5. System nach Anspruch 4, wobei die Rohbilddaten auf einer Vielzahl von Pixeln basieren, wobei die Vielzahl von Pixeln Licht in zwei oder mehr Wellenlängenbändern darstellt, so dass die Rohbilddaten für jedes Pixel einen Intensitätswert umfassen, der die Lichtintensität in einem der zwei oder mehr Wellenlängenbänder darstellt, wobei das Anwenden des Medianfilters für jedes Pixel der Vielzahl von Pixeln die Berechnung eines Medianwertes der Intensität des durch das Pixel dargestellten Lichts und der Intensität des Lichts eines oder mehrerer benachbarter Pixel, die Licht im gleichen Wellenlängenband wie das Pixel darstellen, und die Verwendung des Medianwertes für ein entsprechendes Pixel der entrauschten Rohbilddaten umfasst.
  6. System nach einem der Ansprüche 1 bis 3, wobei die Durchführung des Entrauschens die Anwendung eines Mittelwertfilters zum Entrauschen der Rohbilddaten umfasst.
  7. System nach einem der Ansprüche 1 bis 6, wobei die Handlung des Durchführens des Entrauschens Folgendes umfasst: Anwenden eines Mittelwert- oder Medianfilters zum Entrauschen der Rohbilddaten, Vergleichen des Intensitätswertes eines Pixels der Vielzahl von Pixeln mit einem pixel-spezifischen Intensitätsschwellenwert, nachdem der Mittelwert- oder Medianfilter angewendet wurde, um die Rohbilddaten zu entrauschen, und Anpassen des Intensitätswerts des Pixels, wenn die Intensität höchstens dem pixelspezifischen Schwellenwert entspricht.
  8. System nach einem der Ansprüche 1 bis 7, wobei der pixelspezifische Schwellenwert auf einem für den optischen Abbildungssensor spezifischen Rauschmuster basiert.
  9. System nach Anspruch 8, wobei das für den optischen Abbildungssensor spezifische Rauschmuster ein allgemeines Muster ist, das für optische Abbildungssensoren verschiedener Mikroskope desselben Mikroskoptyps geeignet ist.
  10. System nach Anspruch 8, wobei das für den optischen Abbildungssensor spezifische Rauschmuster ein Muster ist, das unter Verwendung des optischen Abbildungssensors des Mikroskops erzeugt wird,
    und/oder wobei das für den optischen Abbildungssensor spezifische Rauschmuster ein Muster ist, das während der Werkskalibrierung des optischen Abbildungssensors erzeugt wird.
  11. System nach Anspruch 10, wobei das System konfiguriert ist, um das für den optischen Abbildungssensor spezifische Rauschmuster unter Verwendung des optischen Abbildungssensors zu erzeugen.
  12. System nach einem der Ansprüche 1 bis 11, wobei das Entrauschen für Rohbilddaten von Fluoreszenzbildern durchgeführt wird.
  13. Computerimplementiertes Verfahren zur Verarbeitung von Rohbilddaten eines Mikroskops, das Verfahren umfassend:
    Erhalten (210) der Rohbilddaten von einem optischen Bildsensor des Mikroskops,
    wobei die Rohbilddaten auf einem Filtermuster des optischen Bildsensors basieren, die Rohbilddaten auf einer Vielzahl von Pixeln basieren und die Rohbilddaten für jedes Pixel einen Intensitätswert umfassen, der eine Lichtintensität darstellt;
    Durchführen (220) des Entrauschens der Rohbilddaten, um entrauschte Rohbilddaten zu erhalten, wobei der Vorgang des Durchführens des Entrauschens das Vergleichen des Intensitätswerts eines Pixels der Vielzahl von Pixeln mit einem pixelspezifischen Intensitätsschwellenwert und das Anpassen des Intensitätswerts des Pixels umfasst, wenn die Intensität höchstens dem pixelspezifischen Schwellenwert entspricht; und
    Durchführen (250) einer Filtermusterrekonstruktion unter Verwendung der entrauschten Rohbilddaten, um rekonstruierte Bilddaten zu erhalten.
  14. Computerprogramm mit einem Programmcode zur Durchführung des Verfahrens nach Anspruch 13, wenn das Computerprogramm auf einem Prozessor ausgeführt wird.
EP21154120.6A 2020-01-31 2021-01-28 System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops Active EP3859656B1 (de)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP25156597.4A EP4576804A3 (de) 2020-01-31 2021-01-28 System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
DE102020102529 2020-01-31

Related Child Applications (1)

Application Number Title Priority Date Filing Date
EP25156597.4A Division EP4576804A3 (de) 2020-01-31 2021-01-28 System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops

Publications (2)

Publication Number Publication Date
EP3859656A1 EP3859656A1 (de) 2021-08-04
EP3859656B1 true EP3859656B1 (de) 2025-03-05

Family

ID=74418155

Family Applications (2)

Application Number Title Priority Date Filing Date
EP25156597.4A Pending EP4576804A3 (de) 2020-01-31 2021-01-28 System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops
EP21154120.6A Active EP3859656B1 (de) 2020-01-31 2021-01-28 System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP25156597.4A Pending EP4576804A3 (de) 2020-01-31 2021-01-28 System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops

Country Status (1)

Country Link
EP (2) EP4576804A3 (de)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119540311B (zh) * 2025-01-17 2025-05-27 北京林电伟业电子技术有限公司 一种基于机器学习的荧光显微镜校准系统

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1289309B1 (de) * 2001-08-31 2010-04-21 STMicroelectronics Srl Störschutzfilter für Bayermusterbilddaten
US7929798B2 (en) * 2005-12-07 2011-04-19 Micron Technology, Inc. Method and apparatus providing noise reduction while preserving edges for imagers
US8345130B2 (en) * 2010-01-29 2013-01-01 Eastman Kodak Company Denoising CFA images using weighted pixel differences

Also Published As

Publication number Publication date
EP4576804A2 (de) 2025-06-25
EP4576804A3 (de) 2025-07-09
EP3859656A1 (de) 2021-08-04

Similar Documents

Publication Publication Date Title
Monno et al. Single-sensor RGB-NIR imaging: High-quality system design and prototype implementation
US9282305B2 (en) Imaging device and image generation method
US20220005157A1 (en) Method for generating a super-resolution image and related device
US11187657B2 (en) Hyperspectral imaging with fixed pattern noise cancellation
Suman et al. Image enhancement using geometric mean filter and gamma correction for WCE images
KR20190004271A (ko) 매크로포토그래피를 위한 컬러 및 모노 이미지들의 시차 마스크 융합
US9754189B2 (en) Detection device, learning device, detection method, learning method, and information storage device
CN111784686A (zh) 一种内窥镜出血区域的动态智能检测方法、系统及可读存储介质
US10778915B2 (en) Dual-aperture ranging system
WO2020256977A1 (en) Fluorescence imaging with fixed pattern noise cancellation
WO2017086155A1 (ja) 撮像装置、撮像方法、およびプログラム
WO2020256974A1 (en) Fluorescence imaging with fixed pattern noise cancellation
JP5464982B2 (ja) 撮像装置および画像処理方法
WO2020256975A1 (en) Fluorescence imaging with fixed pattern noise cancellation
US20130016198A1 (en) Image processing device, endoscope apparatus, information storage device, and image processing method
EP3859656B1 (de) System, verfahren und computerprogramm zur verarbeitung von rohbilddaten eines mikroskops
US20130155254A1 (en) Imaging apparatus, image processing apparatus, and image processing method
US20160198132A1 (en) Method and apparatus for wide-band imaging based on narrow-band image data
JP2012227758A (ja) 画像信号処理装置及びプログラム
CN211324858U (zh) 内窥镜系统、混合光源、视频采集装置及图像处理器
JP7623397B2 (ja) 顕微鏡システムのためのおよび変換関数を決定するためのシステム、方法およびコンピュータプログラム
JPWO2017104056A1 (ja) 生体情報計測装置、生体情報計測方法および生体情報計測プログラム
US12619060B2 (en) Microscope system and corresponding method
Long et al. A Practical Dental Image Enhancement Network for Early Diagnosis of Oral Dental Disease
CN120837011A (zh) 成像方法和系统及手术机器人系统

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20220204

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230414

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06T0003400000

Ref document number: 602021026991

Country of ref document: DE

Ipc: G06T0003401500

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 25/67 20230101ALI20240720BHEP

Ipc: H04N 25/13 20230101ALI20240720BHEP

Ipc: G06T 5/70 20240101ALI20240720BHEP

Ipc: G02B 21/36 20060101ALI20240720BHEP

Ipc: G06T 5/20 20060101ALI20240720BHEP

Ipc: G06T 3/4015 20240101AFI20240720BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20240917

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602021026991

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250605

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250605

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250606

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1773629

Country of ref document: AT

Kind code of ref document: T

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250707

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250705

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602021026991

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20250305

REG Reference to a national code

Ref country code: CH

Ref legal event code: L10

Free format text: ST27 STATUS EVENT CODE: U-0-0-L10-L00 (AS PROVIDED BY THE NATIONAL OFFICE)

Effective date: 20260114

26N No opposition filed

Effective date: 20251208

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20260126

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20260127

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20260126

Year of fee payment: 6