WO2022144457A1

WO2022144457A1 - A spectral imaging system arranged for identifying benign tissue samples, and a method of using said system

Info

Publication number: WO2022144457A1
Application number: PCT/EP2022/050026
Authority: WO
Inventors: Thomas Nikolajsen; Nicolai Tuelo Pedersen Løbner SHELLER; Filip RENCH-JACOBSEN; Frederik Laust ELBÆK; Knud Poulsen
Original assignee: Hyspec Medtech Ivs
Priority date: 2021-01-04
Filing date: 2022-01-03
Publication date: 2022-07-07
Also published as: DK202170004A1

Abstract

The present invention relates to a spectral imaging system (1) arranged for detecting if a tissue sample (4) is obviously benign or should proceed to histopathological investigation, said imaging system (1) comprises at least one sample container (3) for accommodating a tissue sample (4), at least one light source (5) arranged for sending light through said tissue sample (4), a light detecting device (7) arranged for capturing spectroscopic data based on light transmitted through the tissue sample, and a processing unit (10) arranged for evaluating the captured spectroscopic data and automatically classify whether the tissue sample is obviously benign or should proceed to histopathological investigation. Use of the spectral imaging system (1) ensures that obviously benign tissue samples are sieved out, i.e. said samples (4) need not be sent to the pathology department for further investigation. In this way it is ensured that the pathologist can focus on the more difficult-to-diagnose and/or potentially suspicious tissue samples (4), thereby saving time in the manual examination of histological samples.

Description

A spectral imaging system arranged for identi fying benign tis sue samples , and a method of using said system .

In clinical medicine , tis sue biopsy is a widely used technique for diagnos ing and monitoring diseases , including dif ferent kinds of cancer .

At many pathology department s around the world, a large number of small biopsy samples are examined and evaluated every day . The samples typically go through a histopathological investigation, where the samples are sliced, placed on microscope slides and dyed for microscopic investigation .

In practice, the ma jority of samples turn out to be tumor f ree or benign . As an example can be mentioned that approximately 80% of the one million prostate biopsies performed in the US every year are benign ; suggesting that pathologists are spending 80% of their time analysing benign tis sue .

A further problem i s that the conventionally histopathological methods are characterized by being highly labor intensive, time consuming and expens ive . Accordingly, the investigation of the biops ies constitutes a bottleneck at the pathology departments , and practitioners and surgeons are therefore limited in the number of samples they can collect from a patient .

Furthermore, due to the large number of samples to be investigated test results are often delayed, resulting in undes irable set backs in relevant treatment and/or patients may experience prolonged anxiety awaiting the diagnosi s based on the test result s .

Increased focus has therefore been on computational pathology (digital pathology) which aims at providing quantitative diagnosis of pathological samples , reduction of inter-observer variability among pathologists, and saving time in the manual examination of histological samples .

It has in this respect long been known that the study of light propagation through biological tissues is useful to identify several diseases . As an example can be mentioned that for many years physicians applied a light bulb directly to the surface of the breast and observed the pattern of light transmission on the far side of examined breast. Usually, this approach could only detect the presence of a large mass, and it was not possible to detect if said mass was a fluid filled cysts, a benign tumor or a malignant tumor, see e.g. M. Cutler "Transillumination of the breast” Ann. Surg., 93 (1) , 223-234 (1931) .

However, the properties of the interaction between light and biological tissue has motivated the use of technologies that exploit the obtained information to develop tools for diagnosis support. Special focus has in this respect been on applying fast spectroscopic methods to effectively identify malign tissue .

For instance, development Multispectral imaging (MSI) and Hyper spectral imaging (HSI) to effectively analyze samples for malign tissue has in recent years received increased focus. Many studies have demonstrated the feasibility of using such techniques to detect cancer infected tissue and several comprehensive reviews have summarized the work, see e.g. "Medical hyperspectral imaging : a review" , Journal of Biomedical Optics 19 (1) , 010901 (January 2014) .

Reaches has primarily focused on methods for implementing realtime in vivo screening, especially in order to help surgeons distinguish malignant tissue from benign tissue during surgery. When cancer infected tissue is surgical removed success of the operation relies on ensuring that all malign tissue is removed. The ability of MSI and HSI, to non-invasively, and in real time distinguish benign and malign tissue have increased success rates of cancer operations, see "In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer" , Cancers 2019, 11, 756. However, even though it is an advantage to enable an in-vivo indication of whether or not the tissue is malign, the results is based solely on information obtained from light reflected from the surface of the examined tissue. Thus, if e.g. small island (s) of malign tissue is hidden below the tissue surface, the screening using MSI and HSI will not detect said islands, thereby providing a false negative.

Recently the suggestion to use HSI for analysis of bulk pathological samples has been discussed, see "Tumor margin assessment of surgical tissue specimen of cancer patients using label-free hyperspectral imaging" , Proc. SPIE Int . Soc . Opt. Eng. 2017; 10054. Here it is demonstrated how a HSI system analyzing light, diffusely reflected from the surface of a sample, can be used to effectively distinguish malign and benign tissue regions. Again only data obtained from light reflected from the surface is evaluated, and accordingly false negatives may be obtained, especially if the samples are not sufficiently thin, i.e. in the micrometer range.

Finally, hyperspectral imaging using transillumination has been used for histopathological examination of excised tissue, see "Transillumination hyperspectral imaging for histopathological examination of excised tissue" F. Vasefi et al, Journal of Biomedical Optics 16 (8) , 086014 (August 2011) . However, in this study it is essential that light passing through the sample pass through an angular filter array with a very small acceptance angle. Accordingly only light with an exit angle corresponding to the acceptable angle can be accepted by the filter and thereafter be captured on an image for further evaluation, while scattered light with exit angles outside the speci fic acceptance angle are re jected . Thus , the data obtained using said method is based solely on a limited amount of light pas sing through the sample . Furthermore, the samples still needs to be slices into multiple thin slices and sandwiched between two glas s slides prior to analysis , making this evaluation method time-consuming and expensive .

Thus , is a first aspect of the present invention to provide spectral imaging system and a method of us ing said system, which in a fast and ef fective manner is capable of detecting if a tis sue sample under investigation is obviously benign .

It i s a second aspect of the present invention to provide a spectral imaging system and a method of using said system arranged for evaluating the entire tis sue sample directly, i . e . without any sample pre-treatment step .

It is a third aspect according to the present invention to provide a spectral imaging system and a method of us ing said system that provides quantitative evaluation of ti s sue samples .

It is a forth aspect according to the present invention to provide a spectral imaging system and a method of us ing said system that provides a reduction of inter-observer variability for at least a large number of tis sue samples .

It is a fi fth aspect of the present invention to provide a spectral imaging system which is cost-ef fective and simple and easy to operate .

These and further aspect are achieved according to the present invention by providing a spectral imaging system arranged for detecting if a tis sue sample i s obviously benign or should proceed to histopathological investigation, said imaging system comprises at least one sample container for accommodating a tissue sample, at least one light ssoouurrccee arranged for sending light through said tissue sample, a light detecting ddeevviiccee arranged ffoorr capturing spectroscopic data based on light transmitted through the tissue sample, and a processing unit arranged for evaluating the captured spectroscopic data and automatically classify whether the tissue sample is obviously benign oorr should proceed to histopathological investigation.

Use of the spectral imaging system according to the invention ensures that obviously benign tissue samples aarree sieved out,

1. e . said samples need not be send to the pathology department for further investigation. In this way it is ensured that the pathologist can focus on the more dif f icult-to-diagnose and/or potentially suspicious tissue samples, thereby saving time in the manual examination of histological samples .

There are generally ttwwoo major detection modes relating to spectral imaging depending on the incidence of light within the tissue: light reflection or light transmission. The spectral imaging system according to the present invention is arranged for detecting the light which is transmitted through the tissue sample, i.e. it obtains oonnee oorr more spectroscopic data based primarily on tthhee photophysical processes scattering and absorption of the light which is send through and/or allowed to pass the tissue sample (trans-illumination) .

Scattering of light occurs when there is a spatial variation of the reflective index within the tissue sample. Since a tissue sample comprises a large number of different biological components variations within the tissue may be useful for diagnostic purposes . The absorption of light involves the extraction of energy from light by molecules and can be used as an indication of the molecules' response to light. This also provides information that also can be used for diagnostic purposes .

Thus, when a tissue sample is investigated using the spectral imaging system according to the invention, light will be delivered to the biological tissue and transmitted through said sample. As the light propagates through the tissue, the light will both be scattered, due to the inhomogenecity of the biological structure of said sample, and absorbed due to the presence of some biological components, e.g. hemoglobin, melanin and water. The absorption and scattering characteristics of the tissue will change not only in response to the wavelength used, but also in the presence of abnormalities and/or with the progression of diseases, and the absorbed and transmitted light measured at a number of different wavelengths from the tissue, therefore carry quantitative diagnostic information about tissue physiology, morphology, and composition. It is said information (spectroscopic data) that among others are used to determine if the tissue sample is obviously benign or potentially suspicious .

Within the scope of the present invention the term "obviously benign" relates to tissue samples in which spectroscopic data provides a very high degree of likelyhood/propability that the tissue sample is benign. In the present invention the threshold for determining if a tissue sample is obviously benign is at or above 95%, preferably at least 97%, and even more preferred at least 99%, when compared to a predefined set of tissue samples obtained from a traditional histopathological investigation, and therefore accordingly known to be benign or malignant. All tissue samples falling outside this range, is considered to be potential malignant and should proceed to a conventional histopathological investigation. The spectral imaging system comprises a processing unit arranged for evaluating the captured spectroscopic data and automatically classifies whether the tissue sample is obviously benign or should proceed to histopathological investigation.

Said processing unit may comprise an algorithm arranged for determining the likelihood of a sample being benign, and for classifying the tissue sample as being obviously benign if the likelihood of said tissue sample being benign is at or above a threshold defined by the user e.g. at least 95%, when compared to a predefined set of tissue samples obtained from a traditional histopathological investigation, and therefore accordingly known to be benign or malignant .

The algorithm will typically be a machine learning modules (artificial deep learning neural network, e.g. a conventional neural network (CNN) ) trained to recognize benign and malign patterns in tissue samples obtained from a traditional histopathological investigation. The algorithm preferably takes all the relevant wavelengths into account. E.g. either by using an ensemble network of several machine learning algorithms all being trained on images obtained on different wavelengths, or a CNN build with capacity to have enough input channels needed to train on all wavelengths at the same time. Once sufficiently trained, the machine learning modules can be used to interpret previously unseen spectroscopic data and determine if such spectroscopic data is obviously benign or should proceed to further histopathological investigation.

The machine learning module is considered sufficiently trained when the spectral imaging system has classified a predefined set of tissue samples of at least 1000 as being either benign or malignant, and even more preferred at least 10.000 tissue samples known as being either benign or malignant. In a preferred embodiment the algorithm will be continuously trained over time, and thereby improve, as results from tissue samples that have undergone histopathological investigation are fed back into the system. In order to improve the algorithm even further tissue samples classified as being obviously benign may be randomly selected and send for further histopathological investigation, whereby said results also can be used to train the algorithm even further. Regions on the samples where malign tissue is found may also be labeled/marked, thereby also improving the performance of the algorithm .

In a preferred embodiment the processing unit comprises data transmitting means arranged for informing the user of which tissue sample is considered potentially malignant, and should be investigated further. Such data transmitting means is well known in the art and will not be discussed further in the present application.

The use of imaging spectroscopy in the spectral imaging system according to the present invention provides the advantage that the system integrates conventional imaging and spectroscopy thereby providing both spatial and spectral data from the tissue sample under investigation.

The preferred imaging spectroscopy to be used in the present invention are mult ispectral imaging (MSI) and/or hyperspect ral imaging (HSI) , i.e. the spectral imaging system is a multispectral imaging (MSI) system or a hyperspectral imaging (HSI) system. MSI and HSI are optical spectroscopy imaging modalities, which directly measure the incoming radiance spectra of light, and the technologies are divided into MSI and HSI according to their spectral resolution, number of bands, width and contiguousness of bands, and include the acquisition of image data in both visible and non-visible bands . The invention is however not limited to the MSI and HSI technologies , and the imaging spectros copy may in principle be any relevant spectroscopic method e . g . near infrared reflection and/or transmi s sion spectroscopy, Infrared ( IR) spectroscopic investigation as well as Raman spectroscopy . Furthermore, the spectral imaging technology could be a combination of one or more of the methods mentioned above , the only requirement being that the imaging spectroscopy allows transmis sion of light through the sample, and that spectroscopic data can be captured based on the light pas sing through said sample .

MSI and HSI have the advantages that they are label-free imaging technologies , do not require a contrast agent for obtaining the desired spectroscopic data, and the inventors of the present invention has found that said technologies provides an ob jective as ses sment of whether a tis sue sample is obviously benign or potentially malign .

In order to send light through the ti s sue sample , the spectral imaging system according to the invention comprises a light source arranged for sending light through the tis sue sample . Said light source may in principal be any suitable light source, it is however preferred that the light source covers the full spectral window of interest , e . g . wavelength between 400 nm and 2500 nm, with a high brightnes s , and a f lat , stable spectrum, and without any wavelength "gaps" .

A person skilled in the art will understand that the light source and light detecting device in a preferred embodiment are selected in dependence of each other, and optionally also in relation to the dimens ions of the tis sue sample under investigation . In thi s way it is ensured that the light detecting device is capable of capturing all relevant data f rom the light transmitted through the tis sue sample . However, in a preferred embodiment the spectral density of the light source is at least 1 mV/nm, preferable at least 10 mw/nm, and even more preferred at least 100 mw/nm . The light source may e . g . be a broad spectral device, or one or more lasers having a broadband spectrum covering vi sible, nIR, and SWIR wavelengths from les s than 400 nm to beyond 2500 nm . It is however preferred to have a light source that does not generate any heat which potentially could influence the tis sue sample under investigation, and it i s therefore preferred that the light source is one or more light emitting diodes (LEDs ) as such lamps does not generate and/or radiate heat during operation . One preferred LED is US IHO LIR45A, obtainable f rom Ushio Opto Semiconductor, Inc . having a spectrum from 450 - 1000 nm with a spectra dens ity of >lmW/nm . I f several LEDs are used in the spectral imaging system according to the invention, they may be selected f rom the Ushio EDO high power LED family with wavelength form 365 nm to more than 1000 nm

The spectral imaging system according to the invention may in one embodiment comprise at least two light sources , preferably at least five light sources , and even more preferred at least ten light sources , and wherein each light source covers a separate sub-band, of the spectral coverage of the spectral imaging system . In such an embodiment the spectros copic data may be acquired by turning the respective light sources on sequentially .

The spectral imaging system according to the invention comprises a light detecting device arranged for capturing and/or acquiring spectroscopic information (data ) f rom visible and/or non-vi sible bands from the light that via the light source propagates through the tis sue sample .

Said light detecting device is preferably selected depending on the desired spectros copic data that should be captured by said light detecting device . In a preferred embodiment the light detecting device is a mult ispectral camera, a hyper spectral camera and/or a polarization camera arranged for detecting the degree of polarization and state of polarization for each pixel .

The multi spectral camera is preferably arranged for collecting data in a few and relatively non-contiguous wide spectral bands , typically measured in micrometers , or tens of micrometers . These spectral bands are selected to collect intensity in specifically defined parts of the spectrum and are optimized for certain categories of information most evident in those bands . The inventors of the present invention has found that a combination of spectral bands in both the UV ( 380 nm - 450 nm) , Vis ible ( 450 nm - 700 nm) and Near Infrared ( 700 nm - 1100 nm) is of particular interest e . g . when the light detecting device of the present invention i s a multispectral camera .

When the light detecting device is a hyperspectral camera hundreds of wavelength bands for each pixel is captured, i . e . within a complete spectrum from 350 nm to 1100 nm .

As examples of light sources and light detecting devices , can be mentioned that in one preferred embodiment the light source has a continuous spectrum covering the range from 350 nm - 1000 nm, and the light detecting device is a multi- or hyperspectral camera matching said spectrum . In an alternative embodiment the light source comprises a number light sources with dif ferent wavelengths in the area from 350 nm to 1100 nm, and the light detecting device is a conventional RGB ( red, green , blue ) camera or a B&W (black and white ) camera . A further embodiment the light source is a tunable light source where the wavelength can be varied continuously or dis cretely acros s the entire wavelength range of interest , i . e . between 350 nm and 1100 nm . In this case the light detecting device may also be a conventional RGB or B&W camera . One of the important advantages of both the MSI and HSI technique is that said technologies can acquire a large number of information for each pixels in the image, thereby detecting changes in the ti s sue sample that cannot be identi fied with traditionally gray or color imaging methods .

Each pixel of an HS/MS image represents the light measured by the camera at each specif ic wavelength, creating a set of light measurements which comprise the spectral signature of the tis sue sample . This spectral signature can be understood as a fingerprint of each material in said tis sue sample, and allow dif ferentiation of elements /component s /material s based on the spectral imaging data by using the proces sing unit of the spectral imaging system according to the invention .

The spectral imaging data can be visualized as a three- dimensional cube or a stack of multiple two-dimens ional images . The cube face i s a function of the spatial coordinates and the depth is a function of wavelength, i . e . each picture represents a spectral sub-band of the spectroscopic systems spectral coverage .

A person skilled in the art will understand that the light detecting device need not be a mult ispectral camera or a hyper spectral camera, but could be a di f ferent kind of camera, e . g . a Si-based charge-coupled device (CCD ) camera , and that the hyperspectral cube e . g . may be provided by combining the camera with a light dispersive element such as , but not limited to, a grating a series of optical filters or a Fabry Perot interferometer .

In order to obtain the relevant spectroscopic information the tis sue sample i s placed in a sample container, which is arranged such that the sample preferably is placed between the at least one light source and the light detecting device . Within the context of the present invention, the term "sample container" refers to any container and/or receptacle in which the tissue sample easily can be placed e.g. directly after removal/excision of the tissue sample, and preferably without having to remove the sample container from the spectral imaging system according to the invention. In a preferred embodiment that sample container comprises a bottom and one or more side walls. The side walls will e.g. ensure that the tissue sample is placed and maintained in the correct position during the analysis of said tissue sample, e.g. by preventing the tissue sample from being off-set in relation to the light transmitted from the at least one light source. It is accordingly preferred that the sample container is not a microscopic slide or similar flat device, i.e. a device without side wall ( s ) .

In order to ensure an optimal light intensity the sample container is preferably placed in close proximity to the at least one light source, preferably less than 1 cm. It is furthermore preferred that the light source is aligned (the longitudinal axis of the light source is placed on the same axis as the longitudinal axis of the sample container) with the sample container such that the light source sends light directly into the tissue sample under investigation. Thus, the light source is vertically aligned with the sample container and the light source. Alternatively an optical lens or optical lens system can be used to collect the light from the light source and project it onto the sample.

It is further preferred that the spectral imaging system according to the invention comprises a calibration unit, arranged for providing a reference for the light transmitted through the tissue sample. Said reference comprises, but is not limited to the intensity, brightness, wavelengths of the light emitted from the at least one light source and captured by the light receiving unit. Said calibration unit may be an integrated part of the sample container, or arranged close to, e.g. less than 2 mm, from the sample container, such that the light passing through the sample and the calibration unit are directly comparable. The calibration unit may be one or more aperture (s) allowing light to be transmitted directly (i.e. only through the air) from the light source to the light detecting device. Alternatively the calibration unit may consist of a material with well know optical properties, e.g. similar to the optical properties of tissue. In particular it has proven useful if the material used is paraffin, i.e. the same material as conventionally used for paraffin-embedded tissue samples.

In order to provide a fast and effective way of evaluating if a tissue sample is obviously benign or should proceed to histopathological investigation, it is preferred that the spectral imaging system is arranged such that the tissue sample can be evaluated without any sample pretreatment step, i.e. the tissue sample can be placed in the sample container, that is placed in the spectral imaging system, directly after removal/excision of the tissue sample.

Within the context of the present invention the term "pretreatment step" refers among other to slicing the tissue sample into thin slices, placing the tissue sample on or between glass-slides, dying said tissue sample, and/or any other ways a tissue sample conventionally needs to be processed before it can go through e.g. a histopathological investigation .

It is however important that the tissue sample has a size that can be accommodated in the sample container. The sample container is preferably dimensioned in order to match the characteristics of the light source. In this way it is ensured that sufficient light is transmitted through the tissue sample whereby the relevant spectroscopic data can be obtained/acquired from the tissue sample.

It is however preferred that spectral imaging system according to the invention is arranged for evaluating tissue samples having a thickness of at least 10|lm, preferably at least 100|lm, even more preferably at least 1 mm, and even more preferred at least 5 mm and even more preferred at least 10 mm. Within the context of the present invention thickness is defined as the dimension/thickness of the sample taken along the direct axis from the at least one light source, to the light detecting device, when the tissue sample is placed in the sample container .

The inventors of the present invention have found that when using the spectral imaging system according to the dimensions up to 1 cm³ can be investigated with success, and without any pretreatment .

The spectral imaging system according to the invention thereby provides the advantage that a tissue sample can be taken from a patient, and transferred directly to the sample container for investigation without any human interaction other than the interaction by the person, e.g. surgeon or doctor, removing the tissue sample from the site of interest. A person skilled in the art will understand that if the tissue sample is larger than what can be accommodated in the sample container, the tissue sample may be cut into smaller pieces; however tissue samples are conventionally smaller than 1 cm³. Such a simple division is not considered to be a conventional pretreatment step. If a larger tissue sample is divided into smaller pieces, each piece may advantageously be processed for investigation in the spectral imaging system.

In an alternative embodiment the tissue sample may be embedded in a transparent rigid medium such as e.g. paraffin and/or wax which provides a known scattering/absorption of light in said rigid medium. The inventors of the present invention have found that by embedding the tissue sample in such a transparent rigid medium the absorption and scattering properties of light passing through the tissue sample are "controlled", i.e. the spectroscopic data obtained from such embedded tissue samples may be more reliable. Embedded tissues samples further have the advantage that a reference may be set for lights transmitted through the transparent rigid medium, i.e. without passing through any tissue. In this respect the inventors have found that embedding the sample in paraffin provides a particular advantageous way of illuminating the sample. The optical properties of paraffin are characterized by high scathing and low absorption, and accordingly provide the following advantages :

- light passing through the paraffin provides a good reference

- the strong light scattering effect of paraffin ensures an effective homogenization and distribution of light through the paraffin, ensuring the sample is very evenly illuminated .

Furthermore, embedded tissue samples may be relevant if the tissue sample needs to be preserved for later evaluation, if a reevaluation is desired or for comparative or learning purposes. It is important to stress that the tissue samples are not pre-treated prior to embedding in the transparent rigid medium, i.e. each sample is directly placed/cast in said medium. Embedding tissue samples in e.g. paraffin is a step in a conventional histological investigation, as this enables the pathology to slice the tissue ample in micrometer thin slices. However, the spectroscopic system according to the present invention ensures that the embedded tissue samples can be investigated directed, i.e. without the additional slicing, conventional required. In order to prevent any interference in the spectroscopic data from surrounding light, i.e. light not passing through the tissue sample under investigations the spectral imaging system according to the invention may comprise an optical aperture arranged for discarding surrounding light, i.e. light not passing through the tissue sample. This may for instance be relevant if/when light passes through a part of the sample container not occupied by the tissue sample, e.g. due to irregularities in the tissue samples form/shape, but wherein such surrounding light still is captured by the light detecting device, accordingly influencing the spectroscopic data. Thus, in a preferred embodiment the optical aperture has a dimension which is smaller than the tissue sample under investigation, i.e. light is only allowed to pass through the tissue sample, such that the light detection unit only can capture spectroscopic data from the light that is passed through the tissue sample, thereby effectively preventing any interferences with the surrounding light .

Said optical aperture may either be a part of the sample container, be a screen and/or diaphragm placed between the light source and the sample container, i.e. in the light path from the light source to the tissue sample.

The optical aperture may in a preferred embodiment have a fixed dimension, but in an alternative embodiment the dimensions of the optical aperture can be varied depending on the size of the tissue sample. In a preferred embodiment the optimal aperture is provided by an iris diaphragm which is placed between the light source and the sample container. The opening of the iris diaphragm can easily be adjusted e.g. depending on the dimensions of the tissue sample, to screen away all light from the at least one light source not passing through the sample.

The inventors of the present invention has found that when the tissue sample is embedded in paraffin, the spectral imaging system according to the invention does not need an optical aperture, since the paraf fin itself can ensure that no stray light will hit the light detecting device . Thus , use of paraf fin will ef fectively prevent any interference with the surrounding light .

In order to al so screen the light detecting device f rom external light , e . g . day-light or lamps in the examining room, it is preferred that at least the central parts of the spectral imaging system according to the invention, i . e . the light source, sample container and light detecting device is placed in a housing arranged for blocking out external light , thereby in a fast and ef fective way preventing external light to interfere with the spectroscopic data .

In order to ensure that the tis sue sample in the sample container is aligned with the light source, the optical aperture and/or the light detection unit the sample container may be placed on a motorized XY or XYZ stage, which ensures that the sample can be correctly positioned . It is however preferred that the proces s ing unit is arranged for detecting the location of the tis sue sample in the sample container, and adjust the position of the sample container automatically if the placement i s not optimal .

The spectral imaging system according to the invention may in a further embodiment be arranged for capturing more than one set of spectroscopic data for each tis sue sample, e . g . when the tis sue sample is placed in dif ferent positions relative to the light source, the optical aperture and/or the light detection unit using the motorized XY or XYZ stage .

In one further embodiment the spectral imaging system according to the invention further compri ses at least one polarization filter . Said polarization filter may either be a first polarization filter arranged for polarizing the light from the light source before reaching the tis sue sample or a second polarization filter for controlling the polari zation state of light reaching the light detecting device . It is however preferred that the spectral imaging system according to the invention compri ses both the first and a second polarization filter, in order to determine the deterioration of polarization of the light when going through the sample . The polariz ing filters are typically arranged for cros s polarization microscopy .

The f ilters are arranged with the first polarization filter placed between the at least one light source and the sample container, and the second polarization f ilter placed between the sample container and the light sensitive device .

In one preferred embodiment the first and second polarization filter are aligned with mutually orthogonal polari zation axes .

A person skilled in the art will based on the present application understand that the spectral imaging system may utili ze a polarization camera instead of , or in combination with the first and/or second polari zation f ilter . The polarization camera may e . g . be a XCG-CP510 polarized camera obtainable from Sony . Said camera has a polarization filter depos ited directly on the pixels in the camera .

The spectral imaging system may further comprise further units and/or devices arranged for optimizing the system . As an example can be mentioned that the system may comprise one or more dif fuser plate ( s ) arranged for ensuring that the light transmitted f rom the light source i s homogenized and f ree of spatial variation and/or one or more collimating lens ( es ) arranged for ensuring that the light transmitted through the tis sue sample is accurately aligning/parallel with the camera . Use of a collimating lens further ensures that the light has minimal spread as it propagates into the tis sue sample . The present invention also relates to an automatic tis sue sample system compri sing the spectral imaging system described above . The tis sue sample system may comprise a number of sample containers arranged to move along a proces s line and wherein each sample container will be investigated individually . The number of sample containers may e . g . be arranged in a row, in an array or in a circle, the only requirement being that each sample container i s arranged for accommodating a ti s sue sample which can be individually investigated . In this way the surgeon/doctor can easily place a larger number of tis sue samples in the respective samples containers and either continuous ly or simultaneous ly receive information relating to a large number of sample .

The invention al so relates to a method of using the spectral imaging system according to the invention, and wherein said method comprises the following sequential steps :

- placing a ti s sue sample in the sample container ,

- sending light through the tis sue sample,

- collecting spectroscopic data, and

- determining if the tis sue sample i s obviously benign by comparing the photophys ical data with a predef ined set of tis sue samples obtained from a traditional histopathological investigation .

Thus , using the spectral imaging system according to the invention a tis sue sample can automatically and in les s than a few minutes determine if the tis sue sample under investigation is obviously benign or is potentially suspicions , and accordingly if said sample needs to be investigated further .

This fast and ef ficient detection provides an obvious advantage for a large number of applications . First of all , the number of tis sue samples to be manually investigated is signi ficantly reduced since the pathology department only has to focus on the suspicious samples . This will not only save time and cost for the investigation of the samples , but the patient will also receive a diagnose earlier than using the conventional methods , thereby ensuring a fast initiation of the relevant treatment .

Furthermore, the system according to the invention has an obviously advantage for cancer margin as ses sment during surgery . A complete resection of the tumor is the single most important predictor of patient survival for almost all solid cancers . However a larger number of patients leave the operating room without a complete resection due to positive or close margins . Using the spectral imaging system according to the invention , tis sues samples f rom the margin can easily be investigated and may accordingly be used to navigate cancer resection, thereby aiding in improve the number of complete resections .

The invention will be explained in greater detail below, describing only exemplary embodiments of the spectral imaging system with reference to the drawing, in which

Fig . 1 shows schematically a preferred embodiment of the spectral imaging system according to the invention ,

Fig . 2 shows schematically a second embodiment of the measuring unit of fig . 1 ,

Fig . 3 shows schematically a third embodiment of the measuring unit of fig . 1 ,

Fig . 4 shows s chematically an iri s diaphragm of the measuring unit of fig . 3 ,

Fig . 5 shows schematically a fourth embodiment of the measuring unit of fig . 1 , Fig. 6 shows schematically a fifth embodiment of the measuring unit of fig. 1,

Fig. 7 shows schematically a sixth embodiment of the measuring unit of fig. 1,

Fig. 8 is a flow diagram showing how the algorithm in the form of an artificial deep learning neural network is trained,

Fig. 9 is a flow diagram showing how the algorithm can be used to interpret previously unseen spectroscopic data,

Fig . 10 shows a spectral imaging system according to the invention in more details .

Fig . 11 shows a number of images captured using a spectral imaging system according to the invention, and

Fig . 12 shows schematically the machine learning architecture used in the examples .

The invention will be described below with the assumption that the spectral imaging system a multispectral imaging (MSI) system or a hyperspectral imaging (HSI) system. However, this assumption is not to be construed as limiting, as the system also could be based on e.g. Infrared (IR) spectroscopy or Raman spectroscopy .

Fig. 1 shows schematically a first embodiment of a spectral imaging system 1 according to the invention. The system 1 comprises a measuring unit 2, comprising a sample container 3 for accommodating a tissue sample 4, a light source 5 arranged for sending light 6 through said tissue sample, a light detecting device 7 for capturing spectroscopic data 8 based on light transmitted 9 through the tissue sample 4 and a processing unit 10 arranged for evaluating the captured spectroscopic data 8 and automatically classify whether the tissue sample 4 is obviously benign or should proceed to histopathological investigation.

Depending on the technology used, the light detecting device 7 is a multispectral camera or a hyperspectral camera. The multispectral camera is preferably arranged for collecting data in the UV (380nm - 450nm) , visible (450nm- 700nm) and Near Infrared (700nm- llOOnm) spectra, and the hyperspectral camera is arranged for collecting a large number, i.e. at least 20, wavelength bands for each pixel, and preferably within a complete spectrum from 350 nm to 1100 nm.

The captured spectral imaging data 8 may via a Charged Coupled Device (CCD) of the camera visualized as a three-dimensional cube or a stack of multiple two-dimensional images, and said data is analyzed in the processing unit 10. A CCD is a sensor arranged for capturing light and converts it to digital data that is recorded by the camera. Said processing unit comprises computer 11 with an algorithm 12 arranged for determining the likelihood of a tissue sample 4 being benign, and for classifying the tissue sample as being obviously benign if the likelihood of said tissue sample being benign is larger than a threshold defined by the user e.g. 95%, when compared to a predefined set of tissue samples obtained from a traditional histopathological investigation, and therefore accordingly known to be benign or malignant.

In the embodiment of fig. 1 the algorithm 12 is an artificial deep learning neural network trained to recognize benign and malign patterns in tissue samples obtained from a traditional histopathological investigation.

In order to prevent any interference in the spectroscopic data 8 from surrounding light the sample container 3 comprises an optical aperture 13 which is slightly smaller than the tissue sample 4 under investigation and which thereby ensures that any surrounding light, i.e. light not passing through the tissue sample is discarded. In this way light can only pass through the tissue sample, effectively preventing any interference with the surrounding light in the captured data.

Fig. 2 shows a second embodiment 2a of the measuring unit shown in fig. 1. Said embodiment corresponds basically to the embodiment shown in fig. 1 but the sample container 3 incorporates a calibration unit 14, arranged for providing a reference for the light transmitted from the light source and is captured at the light detecting unit, e.g. the amount of transmitted light and/or the wavelengths of the collected data. The calibration unit 14 in the embodiment shown is a single aperture 15 allowing light to be transmitted directly (i.e. only through the air) from the light source to the camera 7.

Fig. 3 shows schematically a third embodiment 2b of the measuring unit of the invention. Said embodiment corresponds to the embodiment of fig. 1, but in fig. 3 the optical aperture 13 is not part of the sample container 3, but is an adjustable iris diaphragm 16 placed between the light source 5 and the sample container 3, i.e. in the light path 6 from the light source to the tissue sample. The aperture 13' in the iris diaphragm 16 can be adjusted stepless to correspond to the dimensions of the tissue sample 4 such that any light not transmitted through the tissue sample is prevented from reaching the camera 7. In fig. 4 the iris diaphragm is shown with three different sizes of the adjustable aperture, and the aperture 13' of fig. 3 corresponds to the dimensions shown in fig. 4b.

The embodiment of fig. 3 further has the advantage that the sample container is placed on a motorized XYZ stage 17, which ensures that the sample can be correctly positioned relative to the light source 5 and/or the adjustable aperture 13' of the iris diaphragm 16.

The embodiment of fig. 3 may in a fourth embodiment 2c shown in fig. 5 also comprise a calibration unit 14. Said calibration unit is described for fig. 2 and the same principals apply for this embodiment. In fig. 5 the aperture 13' of the iris diaphragm has been adjusted to correspond to the dimensions of the aperture shown in fig. 4a.

Fig. 6 shows a fifth embodiment 2d of the measuring unit of fig. 1. This embodiment corresponds to the embodiment of fig. 5 but further comprises a first polarization filter 18 placed between the at least one light source and the sample container, and the second polarization filter 19 placed between the sample container and the light sensitive device. The first polarization filter 18 is arranged for polarizing the light 6 from the light source 5 before reaching the tissue sample 4 and the second polarization filter 19 is arranged for controlling the polarization state of light 9 reaching the light detecting device 7, whereby the system 1 according to the invention can determine the deterioration of polarization of the light when going through the sample.

In a further embodiment 2e corresponding to the embodiment of fig. 6, the tissue sample 4 is embedded in a transparent rigid medium 20 such as paraffin. This embodiment is shown in fig. 7, The paraffin ensures that the absorption and scattering properties of light passing through the tissue sample are "controlled", i.e. the spectroscopic data 8 obtained from such embedded tissue samples may be more reliable.

Fig. 8 is a flow diagram showing how the algorithm in the form of an artificial deep learning neural network is trained to recognize benign and malign patterns in tissue samples. Said tissue samples are obtained from a traditional histopathological investigation in which a large number of hyper spectral images and their matching classifications obtained through traditional histopathological methods are matched. The machine learning module is considered sufficiently trained when the spectral imaging system has classified a predefined set of tissue samples of at least 1000 as being either benign or malignant, and even more preferred at least 10.000 tissue samples known as being either benign or malignant .

Once sufficiently trained, the machine learning modules can be used to interpret previously unseen spectroscopic data and determine if such spectroscopic data is obviously benign or should proceed to further histopathological investigation. This is shown in fig. 9, in which classification of a tissue sample is performed by the trained deep learning neural network, which returns a classification of whether the sample is obviously benign, or whether the sample requires further investigation.

An example of a spectral imaging system according to the invention is shown in more details in fig. 10. The system comprises a light source 5, consisting of eight high power light emitting diodes (LEDS) 21 placed on a rotating wheel 22. During use the wheel 22 will rotate and sequentially position each of the LEDs 21 underneath a collimating lens 23. Said lens 23 will collimates the light transmitted from the LED and project it onto a diffuser plate 24 located underneath the sample container 3. The purpose of the diffuser plate is to ensure that light from the LEDs is homogenized and free of spatial variation. When a sample 4 placed in the sample container 3 and the light source 5 is turned on, light will be transmitted through the tissue sample and be collected in the camera lens 25. The camera lens is chosen such that it is achromatic and with a field of view large enough to project an image of the entire sample onto a CDD chip of the camera 7. The captured spectral imaging data 8 i s analyzed in the proces sing unit 10 wherein the investigated tis sue sample will be clas s ified as being obviously benign if the likelihood of said tis sue sample being benign is at or larger than a threshold defined by the user e . g . 95% , when compared to a predefined set of ti s sue samples obtained from a traditional histopathological investigation, and therefore accordingly known to be benign or malignant .

Accordingly the spectral imaging system according to the invention provides a fast and ef ficient way of screening a large number of samples , whereby the number of tis sue samples to be manually investigated is signi ficantly reduced since the pathology department only has to focus on the suspicious samples .

EXAMPLE

In order to evaluate the spectral imaging system according to the invention the following experiment were performed, using a spectral imaging system corresponding to the system shown in f ig . 10 .

The light source comprised eight high power light emitting diodes ( LEDs ) with center wavelength spanning from 395 nm to 940 nm . The diodes consi sted of two UV diodes with a center wavelength ( CW) of 395 nm and 425 nm and a full width half maximum spectral width (FWHM) of 20 nm, two diodes with CW of 525 and 600nm and a FWHM of 20 nm and three IR LED with CWs 730 nm, 850 nm and 940 nm respectively with a FWHM of 30 nm . Finally a diode emitting white light covering the entire range from 400-700 nm is also include in the system .

The LEDs are placed on a rotating wheel , arranged with an angle of 40 deg . between each diode , whereby the wheel will rotate during use e . g . by means of a motor, such that each of the LEDs , in the order shown in fig . 11b to Hi , is placed directly under a ti s sue sample accommodated in the sample container . The order of the LEDs must not to be construed as limiting, and the order of the LEDs could in principal be any order .

On the opposite side of the LEDs a RGB CCD camera i s placed, such that when a tis sue sample is placed in the sample container, light will be transmitted through the ti s sue sample and collected in a camera lens placed in front of the camera . The camera has a 1 / 3" CCD chip and is equipped with a compact 25 mm lens from Tameron having a horizontal field view of 11 deg . and a vertical field of view of 8 . 2 deg . At a working distance of 100 mm this yield an image si ze of about 15x20 mm, thereby ensuring that an image of the entire sample can be pro jected onto the CDD chip of the camera .

In order to collimate the light transmitted f rom the respective LED, light will be transmitted first through a collimating lens and then through a dif fuser plate before reaching the tis sue sample . The lens is a piano convex lens from Thorlabs with a focal length of 25 , 4mm and diameter of 1" arranged in such a way as to collimate the output of the LED onto a sheet of Tef lon acting as a dif fuser plate .

The multi spectral imaging system is used by sequentially turning the wheel and recording one image for each of the diodes for each sample . The collection of the recorded images for one sample is referred to as the stack of images . Each diode transmitted light through the tis sue sample for about two second, which was enough time for the camera to capture an image .

In the present experiment s the multispectral imaging system did not comprise an optical aperture s ince the experiment used samples embedded in paraf fin . In this configuration light can only get from the LED side to the camera s ide by pas sing through the paraffin which has similar optical properties as the paraffin eliminating the need for an optical aperture.

In order to provide a normal reflection image of the tissue sample, a white LED with a continuous spectrum covering the range from 400-700nm is placed on the same side of the camera arranged to facilitate the recording of a normal reflection image of the sample, whereby it is possible to compare the obtained/captured data to with the conventional reflection technology normally used for tissue samples.

50 samples with a confirmed diagnosis of being either benign or malign were obtained by embedding the samples in paraffin (in a conventional manner) and thereafter remove the top layer by a planer. Of these 25 of the tissue samples were confirmed to be benign and 25 confirmed to be malign. The paraffin blocks were approximately 3 cm long and 2 cm wide and varied in thickness from a few to about 5 mm.

The spectral imaging system was then used to analyze the samples, and for each sample a stack of two-dimensional images, one image for each LED on the rotating wheel, and one normal reflection image for the white LED placed beside the camera, were captured. An example of the reflection images obtained is shown in fig. Ila, and the images obtained for each diode is shown in fig. 11b - Hi.

The captured images were then used to train a machine learning algorithm to distinguish between malign and benign samples.

The machine learning part of the experiment was implemented by sorting the image stacks according the overall labels "Malign" and "Benign" . Each image stack was tiled into smaller fragments e.g. of 224 x 224 pixels. Each tile from a sample is then labeled malign or benign according to the label of the whole sample . In this experiment the samples were then subsequently split into a training set consi sting of benign and malign samples and a validation set consisting of approximately 20% of the entire sample set .

The machine learning architecture used i s known as an ensemble network, as illustrated in fig . 12 .

For training the network the method of back propagation is used The implementation used is to train a network, in this case a ResNet50 neural net (Convolutional Neural Network ) , for each wavelength individually . For each sample , each tile is fed into the neural network system one by one . The system i s constructed in such a way that each of the images of the tile stack is fed into their own ResNet50 network . Output from all of the resulting 9 networks are then feed into a so-called, fully connected neural network, who then predicts the f inal outcome for each tile, respectively "Malign" or "Benign" .

For each neural network, which is predicting on each wavelength images /tiles the los s (dif ference between the label and the output of the neural network ) is used to train the speci fic network us ing the well-known Adam optimizer function (Adam : A Method for Stochastic Optimization . Diederik P . Kingma and Jimmy Ba ; https : / /arxiv . orq/abs / 1412 . 6980 ) . The ef fect of this is that the network learns to get closer and closer to the label .

When applying this , the inventors have used a dropout functionality of around 50% which means that in order to avoid overf itting, 50 % of the weights was not trained during backpropagation . Number of epochs and which learning rate to apply can change depending of the image samples and the sample size . In this experiment the inventors used the learning rate of 0 . 0001 , because in thi s case it gave the best results measure on the ability to predict benign and malign tiles on, for the software, unknown samples .

After training, the spectral imaging system was tested by feeding each of the tiles from the validation set into the network . By making sure that the same tile (determined by pos ition ) for each image and each wavelength were fed correctly, the inventors could afterwards map out which tiles of the sample the machine learning software predict s is either "Malign" or "Benign" . An example of an image obtained this way is shown in fig . I lk . Here each of the tiles identified as malign are marked with a red square (unbroken line ) , each of the benign samples with a green square (dotted line ) and each of the tiles where the system was not able with suf ficient confidence is left without a square . In this example 100% of tiles from the validation set were clas sified correctly as either benign or malign .

A person skilled in the art will based on the present invention understand that the images may be prepared for subsequent Neural network training in dif ferent ways . As an example can be mentioned, that a principal component analysis (PCA) may be used for removing or reducing the duplication or redundancy in the obtained multispectral images and for compres sing all of the information that is contained in the original multi spectral images into their principal component s . An example of such a PCA image is shown in f ig . 11 j and said image may be used instead of the original data for image analys is and interpretation .

Modif ications and combinations of the above principles and designs are foreseen within the scope of the present invention .

Claims

Claims ,

1. A spectral imaging system (1) arranged for determining if a tissue sample is benign, said imaging system (1) comprises

— at least one sample container (3) for accommodating a tissue sample (4) ,

— at least one light source (5) arranged for sending light through said tissue sample (4) ,

— a light detecting device (7) arranged for capturing spectroscopic data based on light transmitted through the tissue sample (4) , and

— a processing unit (10) arranged for evaluating the captured spectroscopic data and automatically classify whether the tissue sample (4) is obviously benign or should proceed to histopathological investigation.

2. The spectral imaging system (1) according to claim 1, wherein the processing unit (10) comprises an algorithm arranged for determining the likelihood of a tissue sample (4) being benign, and for classifying the tissue sample (4) as being benign if the likelihood of said sample is benign is at or above a threshold defined by the user e.g. 95%, when compared to a predefined set of tissue samples (4) obtained from a traditional histopathological investigation .

3. The spectral imaging system (1) according to claim 2, wherein the algorithm is a machine learning module trained to recognize benign and malign patterns in the predefined set of tissue samples (4) .

4. The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system (1) is a multispectral imaging (MSI) system or a hyper spectral imaging (HSI) system. The spectral imaging system (1) according to any of the preceding claims, wherein the at least one light source (5) covers the full spectral window of interest, such as between 350 nm and 1100 nm. The spectral imaging system (1) according to any of the preceding claims, wherein the at least one light source (5) is a light emitting diode (LED) . The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system (1) comprises at least two light sources (5) , preferably at least five light sources (5) , and even more preferred at least eight light sources (5) , and wherein each light source (5) cover a separate sub-band, and wherein the spectroscopic data is acquired by turning the respective light sources (5) on sequentially. The spectral imaging system (1) according to any of the preceding claims, wherein the light detecting device (7) is a mult ispectral camera. The spectral imaging system (1) according to claim 8, wherein the mult ispectral camera is arranged for collecting spectroscopic data in a few and relatively non-contiguous wide spectral bands, such as in areas between 380 nm and 450 nm, between 450 nm and 700 nm and/or between 700 nm and 1100 nm. The spectral imaging system (1) according to any of the preceding claims 1 - 7, wherein the light detecting (7) device is a hyperspectral camera. The spectral imaging system (1) according to claim 10, wherein the hyperspectral camera is arranged for collecting spectroscopic data in a number of wavelength bands for each pixel within a complete spectrum from 350 nm to 1100 nm. The spectral imaging system (1) according to any of the preceding claims, wherein the light source (5) is aligned with the sample container (3) and the light detecting device (7) . The spectral imaging system (1) according to any of the preceding claims, wherein the sample container comprises a bottom and one or more side walls . The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system comprises a calibration unit (14) , arranged for providing a reference for the light transmitted through the tissue sample ( 4 ) . The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system is arranged for evaluating tissue samples (4) having a thickness of at least 1 mm, and even more preferred at least 5 mm, preferably up to 10 mm, and wherein the thickness of the sample is taken along the direct axis from the at least one light source (5) , to the light detecting device (7) , when the tissue sample (4) is placed in the sample container (3) . The spectral imaging system (1) according to any of the preceding claims, wherein said system (1) is arranged for analyzing a tissue sample (4) , and wherein said tissue sample (4) is embedded in a transparent rigid medium (20) such as paraffin and/or wax which provides a known scattering/absorption of light when transmitted through said rigid medium (20) . The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system comprises an optical aperture (13) arranged for discarding surrounding light . The spectral imaging system (1) according to claim 17, wherein the optical aperture (13) has a dimension which is smaller than a tissue sample (4) in the sample container (3) . The spectral imaging system (1) according to any of the claims 17 and 18, wherein the optical aperture (13) is integrated with the sample container (3) , or is a screen and/or diaphragm placed between the at least one light source (5) and the sample container (3) . The spectral imaging system (1) according to any of the claims 17 - 19, wherein the dimensions of the optical aperture (13) is fixed, or varied depending on the size of the tissue sample (4) placed in the sample container (3) . The spectral imaging system (1) according to any of the preceding claims, wherein the spectral imaging system comprises at least one polarization filter (18,19) , e.g. a first polarization filter (18) arranged for polarizing the light from the light source (5) before reaching the tissue sample (4) and/or a second polarization filter (19) arranged for controlling the polarization state of light reaching the light detecting device (7) . An automatic tissue sample system comprising the spectral imaging system (1) according to any of the claims 1 - 21. The automatic tissue sample system according to claim 22, wherein said automatic tissue sample system comprises a number of sample containers (3) arranged for moving along a process line and wherein each tissue sample in a sample container (3) will be investigated individually. A method of determining if a tissue sample (4) is benign by using the spectral imaging system according any of the claims 1 - 21 or the automatic tissue sample system according to claim 22 or 23, and wherein said method comprises the following sequential steps:

- placing a tissue sample in the sample container (3) ,

- sending light through the tissue sample (4) ,

- collecting spectroscopic data, and

- determining if the tissue sample (4) is benign by comparing the spectroscopic data with a predefined set of tissue samples (4) obtained from a traditional histopathological investigation. The method according to claim 24, wherein the tissue sample (4) is determined as being benign if the likelihood of said sample is benign is at or above a threshold defined by the user e.g. 95%, when compared to a predefined set of tissue samples (4) obtained from a traditional histopathological investigation . The method according to claim 24 or 25, wherein the spectroscopic data from the tissue sample (4) can be collected without any sample pre-t reatment step, such as slicing the tissue sample (4) into thin slices, placing the tissue sample (4) on a glass-slide, between glass-slides, adding contrast agents to the sample and/or dying said tissue sample (4) .