JP2024514493A - Snapshot type high-precision multispectral imaging system and method using multiplexed illumination - Google Patents

Abstract

The present technology generally includes a snapshot type single aperture multispectral imaging device consisting of at least a color camera, a custom designed 8 band optical filter, a motorized zoom lens, and an 8 wavelength multicolor LED illumination system. In addition, the present technology includes a method for designing a time sequence of LED illumination and a combination of LED illumination and an unmixing algorithm to obtain 8 multispectral imaging images. The present device and method can achieve high speed acquisition of multispectral images (less than 100 ms), high speed post-processing without parallax correction calculations by adopting a single aperture system, adjustable field of view, reduced form factor, reduced cost, and high accuracy of spectral information.

Description

Cross-reference to Related Applications This application is filed on March 30, 2021, and is filed under U.S. Provisional Patent Application No. 63/168,151 entitled "Snapshot High Precision Multispectral Imaging System and Method Utilizing Multiplexed Illumination" No. 5,002,101, which application is expressly incorporated herein by reference in its entirety for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT Certain inventions described in this disclosure were made with the Biomedical Advanced Research and Development Agency (BARDA) within the Office of the Assistant Secretary for Preparedness and Response, U.S. Department of Health and Human Services under contract number HHSO100201300022C. It was made with support from the U.S. government granted by the United States. In addition, some of the inventions described in this disclosure may be made by the United States Government under Contract No. W81XWH-17-C-0170 and/or Contract No. W81XWH-18-C-0114 awarded by the United States Government Health Agency (DHA). This was done with the support of The U.S. Government may have certain rights in this invention.

The systems and methods disclosed herein relate to spectral imaging, and more particularly to a multispectral imaging system and method utilizing multiplexed illumination.

The electromagnetic spectrum is a band of wavelengths or frequencies of electromagnetic radiation (e.g., light). From longest to shortest wavelengths, the electromagnetic spectrum includes radio waves, microwaves, infrared light (IR), visible light (i.e., light detectable by structures in the human eye), ultraviolet light (UV), x-rays, and gamma rays. Spectral imaging also refers to a branch of spectroscopy and photography that collects partial or complete spectra of spectral information at various locations in the image plane. Multispectral imaging systems can image multiple spectral bands (usually up to a dozen spectral bands, each in a different spectral region) and collect measurements in the spectral bands for each pixel, and can refer to tens of nm of bandwidth per spectral channel.

Spectral bands may be separated by optical filters and/or multi-channel image sensors (eg, color cameras). Single-aperture multispectral imaging systems using filter wheels have high spectral accuracy, but are limited by low temporal resolution (tens of seconds) due to the slow mechanical rotation of the wheel when switching filters. There is. Multi-aperture spectral imaging systems overcome such limitations by employing polychromatic cameras with multi-bandpass optical filters and utilize unmixing algorithms to achieve some degree of temporal resolution (100 ms Multispectral imaging (MSI) images can be obtained in less than 10 minutes (so-called snapshots). However, such a multi-aperture spectral imaging system still has the problem that parallax correction between a plurality of apertures is complicated.

The multispectral imaging system and multispectral imaging techniques disclosed herein have several features, but any one of them alone cannot achieve the desired characteristics. While certain features of the spectral imaging disclosed herein are briefly described, the following description is not intended to limit the scope of the invention as set forth in the claims below. Those skilled in the art will appreciate how the spectral imaging features disclosed herein have advantages over conventional systems and methods.

In a first aspect of the present technique, a multispectral imaging system includes:
a light source configured to selectively emit light comprising one or more of a set of four or more predetermined frequency bands to illuminate an object;
an image sensor configured to receive a portion of the emitted light that is reflected by the object;
an aperture positioned to pass a portion of the emitted light that is reflected by the object and incident on the image sensor;
a multi-bandpass filter disposed over the aperture and configured to pass light in the four or more predetermined frequency bands;
A memory having instructions stored thereon for generating a multispectral image and performing unmixing; and at least one processor.
The at least one processor performs at least the following steps in accordance with the instructions:
emitting light from the light source in a first subset of two or more of the predetermined frequency bands;
receiving first image data from the image sensor generated based on the reflected light of a first subset of light;
emitting light from the light source in a second subset of two or more of the predetermined frequency bands;
receiving second image data from the image sensor generated based on the reflected light of a second subset of light;
processing the first image data and the second image data to generate at least a first multispectral image and a second multispectral image;
The system is configured to perform spectral unmixing to generate a plurality of images of the object in a single frequency band, each image corresponding to one of the four or more predetermined frequency bands.

In some embodiments, the processor causes the light source to emit light in a third or more subset of two or more frequency bands of the predetermined frequency bands; and a third subset or The image sensor is further configured to receive third or more image data generated based on the reflected light of the further subset of light from the image sensor.

In some embodiments, the light source includes a plurality of light emitting diodes (LEDs), each LED configured to emit light in any one of the four or more predetermined frequency bands. In some embodiments, the at least one processor is further configured to select the frequency bands simultaneously emitted by the light source by controlling activation of individual LEDs of the light source. In some embodiments, the processor controls activation of individual LEDs by controlling electronic switching shutters that selectively pass or block light emitted by each LED.

In some embodiments, the multi-band pass filter includes a plurality of frequency bands for passing light, each frequency band corresponding to any one of the four or more predetermined frequency bands. ing. In some embodiments, the LED and the multi-bandpass filter have the same number of frequency bands, and each frequency band of the LED is aligned to match a corresponding frequency band of the multi-bandpass filter. There is. In some embodiments, the multispectral imaging system further includes a bandpass filter disposed on each individual LED of the plurality of LEDs to confine the light emitted by each individual LED to a precise spectrum. . In some embodiments, the set of four or more predetermined frequency bands includes eight predetermined frequency bands, and the light source includes eight LEDs, each LED having one of the predetermined frequency bands. It is configured to emit light in one of the frequency bands. In some embodiments, the multi-band pass filter includes an 8-band filter configured to pass each of the eight predetermined frequency bands. In some embodiments, the at least one processor causes a third subset of light of the predetermined frequency band to be emitted from the light source according to the instructions, and based on reflected light of the third subset of light. The image sensor is further configured to receive generated third image data from the image sensor. In some embodiments, each of the first subset, second subset, and third subset includes three frequency bands of the predetermined frequency bands. In some embodiments, the predetermined frequency band is defined by a center wavelength ranging from ultraviolet (UV) to short wavelength infrared. In some embodiments, the center wavelengths of the predetermined frequency bands are 420nm, 525nm, 581nm, 620nm, 660nm, 726nm, 820nm, and 855nm, respectively.

In some embodiments, the multispectral imaging system further includes a motorized parfocal zoom lens or a motorized variable focus zoom lens disposed over the aperture, and the one or more processors further configured to control a field of view (FOV) of the multispectral imaging system. In some embodiments, the parfocal zoom lens or the variable focus zoom lens is configured to allow automatic or manual focus adjustment by changing the focal length (FL) of the lens. In some embodiments, adjusting the field of view of the multispectral imaging system does not change the viewing distance between the object and the image sensor. In some embodiments, the one or more processors are further configured to adjust the field of view of the multispectral imaging system by changing a viewing distance between the object and the image sensor. In some embodiments, the processor is further configured to compensate for contrast reduction due to the zoom lens. In some embodiments, the field of view is adjustable over a range of at least 8 cm to 23 cm, such as 8 cm, 9 cm, 10 cm, 11 cm, 12 cm, 13 cm, 14 cm, 15 cm, 16 cm, 17 cm, 18 cm, 19 cm, 20 cm. , 21 cm, 22 cm, or 23 cm, or any two of these lengths as upper and lower limits. In some embodiments, the multispectral imaging system maintains high spatial resolution without reducing the number of samples within the field of view, unlike post-processing digital cropping methods that sacrifice resolution.

In some embodiments, the one or more processors are further configured to visualize the object in natural pseudocolor based on the multispectral image.

In some embodiments, the one or more processors are configured to perform spectral unmixing by determining reflection coefficients of the object using a matrix equation. In some embodiments, the reflection coefficients are determined based at least in part on incident illuminance values, transmission coefficient values of the multiple bandpass filters, and quantum coefficient values of the image sensor for each channel. In some embodiments, the multispectral imaging system is capable of capturing three or more multispectral images in less than 100 milliseconds.

In some embodiments, the object is a tissue region. In some embodiments, the tissue comprises a wound. In some embodiments, the wound comprises a diabetic ulcer, a non-diabetic ulcer, a chronic ulcer, a post-operative wound, an amputation site, a burn, a cancerous lesion, or damaged tissue. In some embodiments, the multispectral imaging system is for use in identifying a tissue classification or identifying a healing score of a tissue, the tissue classification being, for example, live or healthy tissue, dead or necrotic tissue, perfused or non-perfused tissue, or ischemic or non-ischemic tissue, and the healing score of the tissue being, for example, a tendency for at least 50% of the wound to heal after 30 days of standard wound care treatment, or a tendency for at least 50% of the wound to not heal after 30 days of standard wound care treatment. In some embodiments, the multispectral imaging system is for use in imaging a wound, cancer, ulcer, or burn, where the wound, cancer, ulcer, or burn is, for example, a diabetic ulcer, a non-diabetic ulcer, a chronic ulcer, a post-operative wound, an amputation site, a burn, a cancerous lesion, or damaged tissue.

An example of light incident on the filter at various chief ray incidence angles is shown.

1B is a graph showing an example of the transmission efficiency obtained from the filter of FIG. 1A through which light is transmitted at various chief ray angles of incidence.

An example of a multispectral image data cube is shown.

An example is shown of how a particular multispectral imaging technique generates the data cube of FIG. 2A.

2B shows an example of a snapshot-type imaging system capable of generating the data cube of FIG. 2A.

1 shows a cross-sectional schematic diagram illustrating an example optical design of a multi-aperture imaging system with a curved multi-bandpass filter according to the present disclosure. FIG.

3B shows an example of an optical design of optical components constituting one optical path of the multi-aperture imaging system shown in FIG. 3A.

1 shows a schematic diagram of an example of a single-aperture multispectral imaging system according to the techniques of the present invention, and plots showing the performance of each component of the multispectral imaging system.

1 shows an example of a time series of LED illumination when acquiring a multispectral image using the technique of the present invention, and the corresponding detection coefficients for each channel.

1 shows an example of an embodiment embodying a single-aperture multispectral imaging system according to the techniques of the present invention.

3 shows the results of a test with a single aperture system according to the technique of the present invention.

As outlined herein, the present disclosure provides a single-aperture, single-lens, single-camera system that features fast image acquisition speed (snapshots, less than 100 milliseconds), high spectral accuracy, and adjustable field of view. Regarding two-dimensional multispectral imaging (MSI). Additionally, the present disclosure relates to techniques for implementing spectral unmixing to increase the speed of image acquisition from such camera systems. As discussed below, the techniques of the present disclosure overcome problems with current spectral imaging.

Multispectral imaging using multiplexed illumination allows separation of spectral bands, and offers fast switching, robustness, and cost-effectiveness. Currently, such solutions employ multiplexed illumination with overlapping spectra and limited channels. Such multiplexed illumination best captures smoothly varying and well-approximated spectral reflectance (s(λ)) in imaging situations. However, for biomedical applications, highly accurate spectral information is desired due to the somewhat complex spectral reflectance (s(λ)) of biological tissues. To remove such limitations, the crosstalk between spectra needs to be significantly reduced.

The present disclosure provides a snapshot-type high-precision multispectral imaging system and method using multiplexed illumination, which has several features. In some embodiments, the present disclosure provides a multispectral imaging system using a single aperture, single lens, single RGB color camera for two-dimensional multispectral sensing. In some embodiments, the present disclosure provides a multiband optical filter for multispectral filtering. The multiband optical filter in this design may be, for example, an 8-band filter for an 8-band MSI. However, the present disclosure provides a system and method that is not limited to the number specified in this disclosure and may be equally used for MSI using more or less wavelength bands than those specified in this disclosure. In some embodiments, the present disclosure provides a multispectral imaging system using eight wavelengths of light emitting diodes (LEDs) for illumination, each wavelength spectrum overlapping with each MSI band. In some embodiments, a bandpass optical filter with an MSI band matched to each LED is placed in front of each LED. In some embodiments, the present disclosure provides a multispectral imaging system using a motorized zoom lens. To accelerate imaging speed, a spectral unmixing method using LED illumination modulation may be used. Multiple wavelengths of multiple LEDs can be turned on or off in specific combinations and time sequences, which are discussed in more detail below.The unmixing coefficients may be calculated based on the illuminance of the LEDs ( _I0 ), the transmission coefficients of the filters (T%) and/or the detection quantum efficiency (Q%) of the RGB channels of the color camera.

Various embodiments of the multispectral imaging systems and methods of the present disclosure can provide multispectral imaging with improved characteristics, including, but not limited to, one or more of the following:
(a) Because it is a single aperture system, there is no need to use complex algorithms for parallax correction, and no parallax error occurs.
(b) Smaller form factor (e.g., smaller and lighter) and lower cost compared to multi-aperture systems.
(c) It has a fast image acquisition speed (less than 100 ms) and fast post-processing speed, which reduces motion artifacts and allows it to be used for real-time imaging.
(d) Highly accurate spectral information can be obtained by narrow band LED illumination, bandpass optical filters placed in front of the LEDs, and confinement by multiband optical filters.
(e) Since a power zoom lens is used, the field of view can be adjusted according to various applications.
(f) MSI images can be visualized in native pseudocolor.
(g) Multifunctional biomedical imaging can be performed.

The multispectral imaging system of the present invention can achieve a faster image acquisition speed compared to a single aperture multispectral imaging system using a filter wheel with a mechanically rotating filter wheel due to the use of 8-band filters for multi-bandpass optical filtering. At the same time, in some embodiments, spectral information for each MSI band (e.g., channel) is acquired by employing multiple multicolor LEDs with spectra that approximate or exactly overlap the desired MSI band. By utilizing precise spectral illumination and spectral confinement with LED filters and 8-band filters, light leakage outside the spectral band and crosstalk between channels can be avoided, significantly improving spectral accuracy. Each LED can be rapidly cycled on and off to obtain monochromatic illumination using a time-separated system.

In some embodiments, the multispectral imaging system of the present invention can be operated in an ambient or room light environment by acquiring a background image (e.g., without LED illumination) and then (e.g., The background image can be subtracted from the resulting MSI image (exposed to room light and LED lighting). Furthermore, some embodiments employ a multi-channel image sensor (e.g., an RGB camera), simultaneously illuminate three channels of LEDs, and separate the spectral information obtained from the three channels by an unmixing algorithm. The MSI acquisition speed may be further improved by Therefore, the snapshot imaging speed of the techniques of the present disclosure allows for very fast imaging and significantly reduces target motion artifacts, which is advantageous in a variety of biomedical applications.

In some embodiments, the multispectral imaging system and method of the present invention can employ a single aperture, single lens, and single camera configuration, compared to multi-aperture MSI imaging systems, so that the No parallax correction calculations are required, which can require complex algorithms that can significantly slow processing speed. Furthermore, by implementing a configuration consisting of a single aperture, a single lens, and a single camera, misinterpretation of spectral information based on pixels can be avoided. For example, existing multi-aperture MSI imaging systems can utilize image sensor (eg, multiple camera configuration) separation and unmixing algorithms to separate spectral bands. On the other hand, the MSI system and method of the present disclosure may utilize multiplexed illumination (eg, multiple individually controllable multi-color LEDs) and unmixing algorithms to separate spectral bands. The present disclosure provides chronological control of the operation of LED lighting using an electronic switching shutter that is much faster than a mechanical rotating wheel with multiple single-band bandpass filters incorporated into a filter wheel system. As a result, the instantaneous band switching and imaging acquisition speed are similar to multi-aperture MSI imaging systems, there is no need to perform parallax correction calculations, and post-processing speed is fast.

The MSI system and method of the present disclosure can use a band-pass optical filter for the LED illumination and an 8-band optical filter for the detector, so that the detection spectrum can be well confined to the desired band. , crosstalk between each illumination can be avoided, light leakage outside the spectral band can be eliminated, and spectral accuracy can be improved. The present disclosure provides improved benefits over existing systems that include multiplexed lighting in a simple design.

Overview of a Spectral Imaging System FIG. 1A shows an example of a filter 108 placed along an optical path toward an image sensor 110 and light incident on the filter 108 at various ray incidence angles. Rays 102A, 104A, and 106A are shown as rays that pass through filter 108 and then are refracted by lens 112 and incident on sensor 110. Lens 112 may be replaced with other imaging optics such as, but not limited to, mirrors and/or apertures. In FIG. 1A, each beam of light is considered to be broadband, eg, light having a spectral composition over a wide wavelength range, and only light of a specific wavelength is selectively transmitted by filter 108. Three light rays 102A, 104A, 106A enter filter 108 at different angles of incidence. For purposes of illustration, ray 102A is shown as substantially perpendicular incidence to filter 108, ray 104A has a greater angle of incidence than ray 102A, and ray 106A has a greater angle of incidence than ray 104A. has. Each light ray 102B, 104B, 106B passing through the filter exhibits a unique spectrum due to the transmission characteristics of the filter 108 depending on the angle of incidence, and these unique spectra are detected by the sensor 110. This effect of dependence on the angle of incidence causes a shift in the bandpass characteristics of the filter 108, such that the wavelength becomes shorter as the angle of incidence increases. Furthermore, the transmission efficiency of the filter 108 may decrease due to the incident angle dependence, and the spectral shape indicating the bandpass characteristic of the filter 108 may change. Such a combined effect is called angle-of-incidence-dependent spectral transmission. FIG. 1B shows the spectrum of each light ray in FIG. 1A detected by a spectrometer assumed to be present at the location of sensor 110, and shows that as the angle of incidence increases, the spectrum indicating the bandpass characteristic of filter 108 shifts. Explaining. Curve 102C, Curve 104C and Curve 106C show that the center wavelength of the bandpass characteristic is shortened and therefore, in this example, the wavelength of the light emitted from the optical system is shown to be shortened. There is. Furthermore, as shown in this figure, the spectral shape of the bandpass characteristic and the transmittance peak also change depending on the incident angle. In certain consumer applications, image processing can be performed to remove the visible effects of such angle-of-incidence-dependent spectral transmission. However, such post-processing techniques cannot recover accurate information about which wavelength of light actually entered the filter 108. Therefore, the resulting image data may not be usable for certain high precision applications.

Another challenge facing certain existing spectral imaging systems is the time required to acquire a complete spectral image dataset. This will be discussed in connection with Figures 2A and 2B. A spectral image sensor captures the spectral irradiance I(x,y,λ) of a particular scene and collects a three-dimensional (3D) data set commonly referred to as a data cube. FIG. 2A shows an example of a data cube 120 of a spectral image. As shown in this figure, data cube 120 represents three-dimensional image data, of which two spatial dimensions (x and y) correspond to the two-dimensional (2D) surface of the image sensor and a spectral dimension ( λ) corresponds to a specific wavelength range. The size of the data cube 120 can be determined by N _x N _y N _λ , where N _x and N _y are the number of sample points in each spatial dimension (x,y), and N _λ is the spectral axis λ is the number of sample points in the direction. Because the data cube has higher dimensions than currently available 2D detector arrays (e.g., image sensors), typical spectral imaging systems use time-series 2D slices (i.e., planes) of the data cube 120 (hereinafter referred to as " All of the data cube 120 can be imaged using a "scanning" imaging system) or by splitting the data cube 120 during a computational process to obtain multiple two-dimensional data that can be reintegrated into the data cube 120. sample points simultaneously (referred to herein as a "snapshot" imaging system).

FIG. 2B shows an example of how a particular scanning spectral imaging technique generates a data cube 120. Specifically, FIG. 2B shows portions 132, 134, and 136 of the data cube 120 that can be collected in one detector integration time. A point-scanning spectrometer can, for example, image portion 132 that extends across the entire spectral plane λ at one spatial location (x,y). A point-scanning spectrometer can build the data cube 120 by performing multiple integrations for each (x,y) location in the spatial dimensions. A filter wheel imaging system can, for example, image portion 134 that extends across the entire spatial dimensions x and y, but is located in only a single spectral plane λ. A wavelength-scanning imaging system, such as a filter wheel imaging system, can build the data cube 120 by performing multiple integrations over the spectral plane λ. A line-scanning spectrometer can, for example, image portion 136 that extends across the entire spectral dimension λ and across one of the spatial dimensions (x or y), but is located at only one point in the other spatial dimension (y or x). A line-scan spectrometer can construct a data cube 120 by performing multiple integrations over the position of one of the spatial dimensions (y or x).

In applications where both the target object and the imaging system are stationary (or relatively stationary during the exposure time), the scanning imaging system described above has the advantage of providing a high resolution data cube 120. Line-scan and wavelength-scanning imaging systems may provide these benefits by using the entire area of the image sensor to capture each spectral or spatial image. However, if there is movement in the imaging system and/or object between the first exposure and the next exposure, artifacts may occur in the resulting image data. For example, the same (x,y) location in data cube 120 may actually represent a different physical location on the imaged object in the spectral dimension λ. Such artifacts may introduce errors in subsequent analysis and/or may necessitate alignment (e.g. if a particular (x,y) location has the same physical It becomes necessary to adjust the position of the spectral dimension λ so that it corresponds to the target position).

In contrast, a snapshot imaging system 140 can capture the entire data cube 120 in one integration time or exposure, thus avoiding the effects of motion on image quality. FIG. 2C shows an example of an image sensor 142 and an optical filter array 144 (such as a color filter array (CFA)) that can be used to create a snapshot imaging system. In this example, the color filter array (CFA) 144 is a repeating pattern of color filter units 146 arranged on the surface of the image sensor 142. This method of obtaining spectral information can also be called a multispectral filter array (MSFA) or a spectrally resolved detector array (SRDA). In the example shown in this figure, the color filter unit 146 includes various color filters arranged in a 5×5 array, thereby generating 25 spectral channels in the resulting image data. The CFA can use various color filters to split the incoming light into bands for each color filter and direct the split light to the photoreceptors of each color on the image sensor. In this way, for a given color 148, only one of the 25 photoreceptors actually detects a signal indicative of light of that wavelength. Thus, with such a snapshot imaging system 140, 25 color channels may be generated in one exposure, but the amount of measurement data for each color channel is less than the total output of the image sensor 142. In some embodiments, the CFA may include a filter array (MSFA) and/or a spectrally resolved detector array (SRDA), and/or may include conventional Bayer filters, CMYK filters, or other absorptive or interferometric filters. One type of interferometric filter is a thin film filter array arranged in a grid, with each grid element corresponding to one or more sensor elements. Another type of interferometric filter is the Fabry-Perot filter. Nano-etched Fabry-Perot interference filters typically exhibit bandpass characteristics with a full width at half maximum (FWHM) of about 20-50 nm (e.g., 20 nm, 21 nm, 22 nm, 23 nm, 24 nm, 25 nm, 26 nm, 27 nm, 28 nm, 29 nm, 30 nm, 31 nm, 32 nm, 33 nm, 34 nm, 35 nm, 36 nm, 37 nm, 38 nm, 39 nm, 40 nm, 41 nm, 52 nm, 53 nm, 44 nm, 45 nm, 46 nm, 47 nm, 48 nm, 49 nm, or 50 nm, or a range defined by any two of these wavelengths), but are advantageous in that they can be used in some embodiments due to their low roll-off from the center wavelength of the filter's passband to the stopband. Additionally, such filters exhibit a low optical density (OD) in the stopband, which can improve sensitivity to light outside the passband. Due to these combined effects, these particular filters exhibit sensitivity to spectral regions that would be blocked by the high roll-off of similar FWHM high optical density interference filters constructed from multiple thin film layers deposited by coating deposition techniques such as evaporation or ion beam sputtering. In embodiments using dye-based CMYK or RGB-based (Bayer) filters, low spectral roll-off and large FWHM of the passbands of the individual filters are preferred, allowing unique spectral transmission for each wavelength across the observed spectrum.

Therefore, the data cube 120 obtained by the snapshot imaging system has one of the following two characteristics that can be problematic in high-precision imaging applications. The first problem is that the _data cube 120 obtained by a _snapshot imaging system is The resolution will be lower than the data cube 120 obtained by a scanning imaging system with an image sensor. A second problem is that in the data cube 120 obtained by a snapshot imaging system, numerical interpolation for a particular (x,y) position allows the magnitudes _{of N} may be the same size as y). However, if interpolation is done in the generation of the data cube, it means that the particular number in the data cube is not the actual value of the wavelength of the incident light on the sensor, but an estimate of the actual value based on the surrounding values. .

Another existing component used in multispectral imaging with a single exposure is the multispectral imaging beam splitter. In such imaging systems, a beam splitter cube splits the incoming light into different color bands, each of which is observed by separate image sensors. The beam splitter design can be modified to adjust the spectral bands measured, but it is not easy to split the incoming light into more than four beams without compromising the performance of the imaging system. Thus, four spectral channels appears to be the practical limit of this approach. A closely related method splits the light using thin film filters instead of bulky beam splitter cubes/prisms, but the cumulative transmission attenuation and spatial limitations of multiple successive filters limit this approach to about six spectral channels.

In some embodiments, the aforementioned challenges, among others, can be addressed by the spectral imaging system and associated image data processing techniques disclosed herein that utilize multiplexed illumination, multiple bandpass filters that selectively transmit the illumination rays, and color channels of an RGB camera. This particular configuration can achieve all of the design goals of fast imaging speed, high resolution images, and high fidelity of the detection wavelength. Thus, the optical design and associated image data processing techniques disclosed herein can be used in a portable spectral imaging system and/or can be used to image moving subjects and obtain data cubes suitable for high precision applications (e.g., clinical tissue analysis, biometrics, and episodic clinical signs). Such high precision applications that can be achieved using one or more embodiments described herein include diagnosis of pre-metastatic stages (0-3) of basal cell carcinoma, squamous cell carcinoma, and malignant melanoma; classification of the severity of burns or wounds in skin tissue; identification of necrotic or ischemic tissue and its margins, such as by comparison with healthy or normal skin; or tissue diagnosis or severity determination of peripheral vascular disease or diabetic foot ulcers. Thus, the snapshot spectral acquisition and small form factor described in some embodiments allows the present invention to be used in clinical settings dealing with transient events, including, for example, diagnosing various types of retinopathies (e.g., non-proliferative diabetic retinopathy, proliferative diabetic retinopathy, and age-related macular degeneration), as well as imaging of pediatric patients with high levels of motion. Thus, one skilled in the art will appreciate that the use of the systems disclosed herein represents a significant technological advance over conventional implementations of spectral imaging.

Various aspects of the disclosure are discussed below in connection with specific examples and embodiments. These examples and embodiments are for illustrative purposes and are not intended to limit this disclosure. Although the examples and embodiments described herein focus on specific calculations and algorithms for illustrative purposes, those skilled in the art will appreciate that these examples are for illustrative purposes only and that the present invention It will be understood that this does not limit the invention. For example, although some examples are given with respect to multispectral imaging, the single aperture imaging systems and associated filters and multiplexed illumination disclosed herein may be used to perform hyperspectral imaging in other implementations. Can be configured. Additionally, while specific examples are presented as achieving advantages in hand-held applications and/or applications with moving objects, the imaging system designs and processing techniques disclosed herein are It will be appreciated that this can provide a highly accurate data cube suitable for fixed imaging systems and/or analysis of relatively stationary objects.

Electromagnetic Spectral Ranges and Image Sensor Overview Reference is made herein to particular colors or portions of the electromagnetic spectrum, and these wavelengths are hereinafter described as defined by ISO 21348, "Definitions of Irradiance Spectral Types." As will be described in more detail below, in certain imaging applications, wavelength ranges of particular colors may be passed collectively through particular filters.

Electromagnetic radiation in the wavelength range of 760 nm to 380 nm or about 760 nm to 380 nm is usually considered to be the "visible" spectrum, i.e., the portion of the spectrum that is perceptible by the color receptors of the human eye. In the visible spectrum, red light is usually considered to have a wavelength of at or about 700 nm, or to be in the range of 760 nm to 610 nm or about 760 nm to 610 nm. Orange light is usually considered to have a wavelength of at or about 600 nm, or to be in the range of 610 nm to 591 nm or about 610 nm to 591 nm. Yellow light is usually considered to have a wavelength of at or about 580 nm, or to be in the range of 591 nm to 570 nm or about 591 nm to 570 nm. Green light is usually considered to have a wavelength of at or about 550 nm, or in the range of 570 nm to 500 nm, or about 570 nm to about 500 nm. Blue light is usually considered to have a wavelength of at or about 475 nm, or in the range of 500 nm to 450 nm, or about 500 nm to about 450 nm. Violet (magenta) light is usually considered to have a wavelength of at or about 400 nm, or in the range of 450 nm to 360 nm, or about 450 nm to about 360 nm.

Looking outside the visible spectrum, infrared (IR) refers to electromagnetic radiation that has longer wavelengths than visible light and is usually invisible to the human eye. IR has wavelengths ranging from about 760 nm or the nominal lower limit of the red visible spectrum at 760 nm to about 1 mm or 1 mm. Within this range, near-infrared (NIR) refers to the part of the spectrum adjacent to the red range, which ranges in wavelength from about 760 nm to about 1400 nm or from 760 nm to 1400 nm.

Ultraviolet (UV) light refers to the portion of electromagnetic radiation with wavelengths shorter than visible light and is generally invisible to the human eye. UV has wavelengths ranging from about 40 nm or the nominal lower limit of the visible spectrum at violet at 40 nm to about 400 mm. Within this range, near ultraviolet (NUV) refers to the portion of the spectrum adjacent to the violet range and has a wavelength range of about 400 nm to about 300 nm or 400 nm to 300 nm, mid ultraviolet (MUV) has a wavelength range of about 300 nm to about 200 nm or 300 nm to 200 nm, and far ultraviolet (FUV) has a wavelength range of about 200 nm to about 122 nm or 200 nm to 122 nm.

The image sensors described herein can be configured to detect electromagnetic radiation in any of the above ranges, depending on the particular wavelength range that is appropriate for a particular application. Typical silicon charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) sensors have spectral sensitivity that spans the entire visible spectrum, but also most of the near-infrared (IR) spectrum and parts of the UV spectrum. In some implementations, back-illuminated or front-illuminated CCD or CMOS arrays can additionally or alternatively be used. For applications requiring high signal-to-noise ratios and scientific-grade measurements, some implementations can additionally or alternatively use scientific complementary metal-oxide semiconductor (sCMOS) cameras or electronically multiplied CCD cameras (EMCCDs). In other implementations, sensors known to operate in a particular color range (e.g., short-wave infrared (SWIR), mid-wave infrared (MWIR), or long-wave infrared (LWIR)) and corresponding optical filter arrays can additionally or alternatively be used, depending on the intended application. These sensors and optical filter arrays may additionally or alternatively include cameras based on detector materials such as indium gallium arsenide (InGaAs) or indium antimony (InSb), or cameras based on microbolometer arrays.

Image sensors used in the multispectral imaging techniques disclosed herein may be used in conjunction with optical filter arrays, such as color filter arrays (CFAs). Some types of CFAs split the incident light into red (R), green (G), and blue (B) visible regions, and direct the split visible light to dedicated red, green, or blue regions on the image sensor. The photodiode can be guided to the photoreceptor. A common example of a CFA is the Bayer pattern, which is a color filter array with RGB color filters arranged in a specific pattern on a rectangular grid of photodetectors. The Bayer pattern consists of 50% green, 25% red and 25% blue, with rows of repeated red and green color filters and repeated rows of blue and green color filters. The rows are arranged alternately. Some types of CFA (such as those for RGB-NIR sensors) can also split NIR light and direct the split NIR light to dedicated photodiode photoreceptors on the image sensor.

Therefore, the wavelength range of the filter components of the CFA may determine the wavelength range represented by each image channel in the captured image. Accordingly, in various embodiments, the red channel of the image may correspond to the red wavelength region of the color filter and may include a portion of yellow light and orange light, from about 570 nm to about 760 nm or 570 nm. ~760nm range. In various embodiments, the green channel of the image may correspond to the green wavelength region of the color filter and may include a portion of yellow light, ranging from about 570 nm to about 480 nm or from 570 nm to 480 nm. be. In various embodiments, the blue channel of the image may correspond to the blue wavelength region of the color filter and may include a portion of violet light, ranging from about 490 nm to about 400 nm or from 490 nm to 400 nm. be. Those skilled in the art will appreciate that the exact starting and ending wavelengths (or portions of the electromagnetic spectrum) that define the colors of a CFA (e.g., red, green, and blue) may vary depending on the implementation of the CFA. You will be able to fully understand this.

Additionally, typical visible light CFAs transmit light outside the visible spectrum. Therefore, many image sensors limit their sensitivity to infrared radiation by placing a thin film infrared reflective filter in front of the sensor that blocks infrared wavelengths but allows visible light to pass. However, in some of the imaging systems disclosed herein, the thin film infrared reflective filter may be omitted to allow infrared light to pass through. Accordingly, a red channel, a green channel and/or a blue channel may be used to collect infrared wavelength bands. In some implementations, the blue channel may be used to collect specific NUV wavelength bands. At each wavelength in the stacked spectral images, the red, green, and blue channels exhibit different spectral responses associated with each channel having a unique transmission efficiency, so we used a known transmission profile. may provide a unique weighting response for each spectral band before unmixing. For example, this weighted response may include the known transmission responses of the red, blue, and green channels in the infrared and ultraviolet wavelength regions, thereby allowing bands from these wavelength regions to Each channel can be used to collect data.

Selectively extracting specific bands of light incident on the image sensor by placing additional color filters in front of the CFA along the optical path toward the image sensor, as detailed further below. Can be done. The color filters disclosed herein are a combination of (thin film) dichroic filters and/or absorption filters, or a single dichroic filter, and/or a single absorption filter. Some of the color filters disclosed herein are bandpass filters that pass frequencies within a particular region (in the passband) but block (attenuate) frequencies outside that region (in the rejection region). Good too. Some of the color filters disclosed in this specification may be multi-bandpass filters that pass a plurality of discontinuous wavelength regions. These "frequency bands" may have narrower passbands, greater attenuation in the stop region, and steeper spectral roll-offs than the wide color range of the CFA filter. Note that rolloff is defined as the steepness of the spectral response at the transition from the passband to the stopband of the filter. For example, the color filters disclosed herein can cover a passband from about 20 nm to about 40 nm or from 20 nm to 40 nm. This particular configuration of the color filter may determine the wavelength band actually incident on the sensor, which may improve the accuracy of the imaging techniques disclosed herein. The color filters described herein can be configured to selectively block or pass certain bands of electromagnetic radiation within the above-mentioned ranges, depending on the particular wavelength band appropriate for a particular application. can.

The term "pixel" is used herein to describe the output produced by the elements of a two-dimensional detector array. In contrast, the photodiode, i.e. one photosensitive element, of the two-dimensional detector array converts photons into electrons via the photoelectric effect, which then converts the electrons into signals that can be used to determine pixel values. It behaves as a transducer that can. One element of a data cube may be referred to as a "voxel" (eg, a volumetric element). A “spectral vector” refers to a vector that represents spectral data (eg, the spectrum of light received from a particular point in object space) at a particular (x,y) location in the data cube. One horizontal plane of the data cube (eg, an image representing one spectral dimension) is referred to herein as an "image channel." In certain embodiments described herein, spectral video information may be captured and the resulting data dimensions are assumed to be in the form of a "hypercube" denoted N _x N _y N _λ N _t (where N _t is the number of frames taken during the video sequence).

Overview of an Example Imaging System FIG. 3A shows a schematic diagram of an example multi-aperture imaging system 200 with curved multi-bandpass filters according to the present disclosure. The schematic diagram shown includes a first image sensor area 225A (photodiodes PD1-PD3) and a second image sensor area 225B (photodiodes PD4-PD6). The photodiodes PD1-PD6 may be, for example, photodiodes formed on a semiconductor substrate (e.g., a CMOS image sensor). Typically, each photodiode PD1-PD6 may be a single unit of some material, semiconductor, sensor element, or other device capable of converting incident light into an electrical current. This diagram shows only a small portion of the entire multi-aperture imaging system for purposes of illustrating the structure and operation of the multi-aperture imaging system, and it will be appreciated that in an implementation, the image sensor area may include hundreds or thousands of photodiodes (and corresponding color filters). The first image sensor area 225A and the second image sensor area 225B may be implemented as separate sensors or as separate areas on the same image sensor, depending on the implementation. Although FIG. 3A shows two apertures and corresponding optical paths and sensor areas, it will be appreciated that the optical design principles illustrated in FIG. 3A can be extended to designs including three or more apertures and corresponding optical paths and sensor areas, depending on the implementation.

Multi-aperture imaging system 200 includes a first aperture 210A that provides a first optical path toward a first sensor area 225A, and a second aperture that provides a first optical path toward a second sensor area 225B. Including 210B. These apertures may be adjustable to increase or decrease the brightness of the light reflected in the image, or alternatively, by adjusting these apertures, the exposure time for a particular image can be adjusted. The brightness of the light incident on the image sensor area may be changed so that the brightness of the light incident on the image sensor area does not change. These apertures may be located at any location along the optical axis of the multi-aperture system as determined to be reasonable by one skilled in the art of optical design. The optical axis of the optical component disposed along the first optical path is indicated by a dashed line 230A, and the optical axis of the optical component disposed along the second optical path is indicated by a dashed line 230B. It is well understood that it does not represent 200 physical structures. The optical axis 230A and the optical axis 230B are separated by a distance D, which can be a parallax between the image taken by the first sensor area 225A and the image taken by the second sensor area 225B. . "Disparity" refers to the distance between two corresponding points on the left and right sides (or top and bottom) of a stereophotographic image, such that the same physical point in object space appears at different positions in each image. The processing techniques used to correct for this parallax will be described in more detail below.

Optical axis 230A and optical axis 230B pass through the center C of their respective apertures. Other optical components can also be arranged around these optical axes (for example, the rotational symmetry points of the optical components can be arranged along this optical axis). For example, the first curved multi-band pass filter 205A and the first imaging lens 215A can be arranged around the first optical axis 230A, and the second curved A multi-bandpass filter 205B and a second imaging lens 215B can be arranged.

As used herein, the terms "on" and "above" when discussing the placement of optical elements refer to the location of a particular structure (e.g., a color filter or lens) through which light entering the imaging system 200 from the object space passes before reaching (or entering) another structure. To illustrate this, along a first optical path, the curved multi-bandpass filter 205A is located above the aperture 210A, which is located above the imaging lens 215A, which is located above the CFA 220A, which is located above the first image sensor area 225A. Thus, light entering from the object space (e.g., the physical space to be imaged) first passes through the first curved multi-bandpass filter 205A, then through the aperture 210A, then through the imaging lens 215A, then through the CFA 220A, and finally enters the first image sensor area 225A. A second optical path (e.g., an optical path including curved multi-bandpass filter 205B, aperture 210B, imaging lens 215B, CFA 220B, and second image sensor area 225B) may be similarly arranged. In another implementation, apertures 210A, 210B and/or imaging lenses 215A, 215B may be located above curved multi-bandpass filters 205A, 205B. Additionally, another implementation may not use a physical aperture and may rely on a clear aperture of an optical component to control the brightness of the light imaged onto sensor area 225A and sensor area 225B. Thus, lenses 215A, 215B may be located above apertures 210A, 210B and curved multi-bandpass filters 205A, 205B. In this implementation, the apertures 210A and 210B may be located above the lenses 215A and 215B, or the apertures 210A and 210B may be located below the lenses 215A and 215B, as may be determined by one of ordinary skill in the art of optical design.

The first CFA 220A placed above the first sensor area 225A and the second CFA 220B placed above the second sensor area 225B function as filters that selectively pass specific wavelengths. It is possible to divide the incident light in the visible region into a red region, a green region, and a blue region (represented by symbols R, G, and B, respectively). The light is "split" by passing only certain selected wavelengths through each color filter of the first CFA 220A and the second CFA 220B. The divided light is received by a dedicated red diode, green diode, or blue diode on the image sensor. Although red filters, blue filters and green filters are commonly used, in other embodiments various color filters can be used depending on the color channels required for the captured image data, e.g. Similar to RGB-IR CFA, filters that selectively pass ultraviolet, infrared, or near-infrared radiation can be used.

As shown in FIG. 3A, the filters of the CFA are positioned above single photodiodes PD1-PD6. FIG. 3A also shows an example of a microlens (designated ML) that may be formed or otherwise disposed on each color filter to focus incident light onto the active detector area. Alternative implementations may have multiple photodiodes (e.g., a collection of two, four, or more adjacent photodiodes) below a single filter. In the example shown, photodiodes PD1 and PD4 are located below a red filter and therefore output pixel information for a red channel, photodiodes PD2 and PD5 are located below a green filter and therefore output pixel information for a green channel, and photodiodes PD3 and PD6 are located below a blue filter and therefore output pixel information for a blue channel. Additionally, as described in more detail below, the particular color channel output by a given photodiode can be further narrowed based on the active light source and/or the particular frequency bands passed through the multiple bandpass filters 205A, 205B, allowing different image channel information to be output from a given photodiode with different exposures.

The imaging lenses 215A, 215B can be shaped to focus the image of the object scene on the sensor area 225A, 225B. Each imaging lens 215A, 215B is not limited to a single convex lens as shown in FIG. 3A, but may be composed of as many optical elements and surfaces as needed to form an image, and various types of imaging lenses or lens assemblies, either commercially available or custom designed, can be used. Each optical element or lens assembly can be formed or coupled to each other to be housed in series or stacked using an opto-mechanical barrel with a retaining ring or bezel. In some embodiments, the optical element or lens assembly can include one or more coupled lens groups, which can include two or more optical components bonded or otherwise coupled together. In various embodiments, the multiple bandpass filters described herein may be placed in front of a lens assembly of a multispectral imaging system, in front of a single lens of a multispectral imaging system, behind a lens assembly of a multispectral imaging system, behind a single lens of a multispectral imaging system, inside a lens assembly of a multispectral imaging system, inside a group of combined lenses of a multispectral imaging system, directly on the surface of a single lens of a multispectral imaging system, or directly on the surface of an element of a lens assembly of a multispectral imaging system. Additionally, apertures 210A and 210B may be eliminated and lenses 215A and 215B may be of the type typically used in digital single-lens reflex (DSLR) or mirrorless camera photography. Additionally, lenses 215A and 215B may be of the type used in machine vision using a C-mount or S-mount thread for attachment. Focus adjustment can be performed, for example, by moving the imaging lenses 215A, 215B relative to the sensor areas 225A, 225B, or by moving the sensor areas 225A, 225B relative to the imaging lenses 215A, 215B, based on manual focus, contrast-based autofocus, or other suitable autofocus techniques.

Each multi-bandpass filter 205A, 205B can be configured to selectively pass multiple lights in a narrow frequency band, for example, in some embodiments multiple lights in a frequency band from 10 to 50 nm ( In other embodiments, it may be configured to selectively pass a wider or narrower frequency band of light. As shown in FIG. 3A, both multi-bandpass filters 205A and 205B can pass the frequency band λc ("common frequency band"). In implementations with more than two optical paths, each multi-bandpass filter can pass this common frequency band. In this manner, each sensor area captures image information in the same frequency band (a "common channel"). As discussed in more detail below, this image information obtained on this common channel can be used to align the set of images taken by each sensor region. Some implementations may have one common frequency band and corresponding common channel, or may have multiple common frequency bands and corresponding common channels.

Each of the multi-bandpass filters 205A, 205B can be configured to selectively pass one or more unique frequency bands in addition to the common frequency band λc. In this manner, the imaging system ₂₀₀ can collectively image a greater number of different spectral channels by the sensor areas 205A, 205B than can be imaged by a single sensor area. This aspect is illustrated in FIG. 3A as multi-bandpass filter 205A passing unique frequency band λu1 and multi-bandpass filter 205B passing unique frequency band _λu2 , where _λu1 and _λu2 represent different frequency bands. Although shown passing two frequency bands in this figure, the multi-bandpass filters of the present disclosure can each pass more than one frequency band at a time. For example, as described below in connection with FIGs. 11A and 11B, in some implementations, each of the multi-bandpass filters can pass four frequency bands. In various embodiments, more frequency bands may be passed. For example, a four camera implementation may include a multi-bandpass filter configured to pass eight frequency bands, in some embodiments the number of frequency bands may be, for example, four, five, six, seven, eight, nine, ten, twelve, fifteen, sixteen or more.

The multi-bandpass filters 205A, 205B have a curvature selected to reduce the incident angle-dependent spectral transmission in the respective sensor regions 225A, 225B. As a result, when receiving narrowband illumination from the object space, each photodiode (e.g., a color filter that covers the sensor area and passes that wavelength The filter is believed to receive light at substantially the same wavelength without the wavelength shift seen in photodiodes placed near the edge of the sensor, as described above in connection with FIG. 1A. With such a configuration, more accurate spectral image data can be generated than when using a flat filter.

FIG. 3B shows an example of the optical design of optical components that constitute one optical path of the multi-aperture imaging system shown in FIG. 3A. More specifically, FIG. 3B shows a custom achromatic doublet 240 that can be used as a multi-bandpass filter 205A, 205B. This custom achromatic doublet 240 passes light through the housing 250 and onto the image sensor 225. Housing 250 may include openings 210A, 210B and imaging lenses 215A, 215B, as described above.

The achromatic doublet 240 is configured to correct optical aberrations due to the incorporation of the required surfaces for the multi-bandpass filter coatings 205A, 205B. The achromatic doublet 240 shown in this figure includes two lenses that can be made of glass or other optical materials with different dispersions and refractive indices. In other implementations, three or more lenses may be used. These achromatic doublet lens designs incorporate the multi-bandpass filter coatings 205A, 205B on the curved front surface 242, and incorporate the optical surfaces of the curved single lens onto which the filter coatings 205A, 205B are deposited to cancel optical aberrations, and the combined effect of the curved front surface 242 and curved back surface 244 of the achromatic doublet 240 limits the refractive or light gathering power, so that these lenses alone, housed in the housing 250, can constitute the primary elements for light gathering. The achromatic doublet 240 can thus contribute to improving the accuracy of image data captured by the imaging system 200. These individual lenses can be mounted adjacent to one another, e.g., by bonding or cementing the individual lenses together, and shaped so that the aberrations of one lens are offset by the aberrations of the other lens. The curved front surface 242 or the curved back surface 244 of the achromatic doublet 240 can be coated with multiple bandpass filter coatings 205A, 205B. Alternative doublet designs may be implemented in the systems described herein.

Further variations of the optical designs described herein may be implemented. For example, in some embodiments, a single lens or other optical single lens, such as a convex (positive) or concave (negative) lens as shown in FIG. 3A, may be included in the optical path instead of the doublet 240 shown in FIG. 3B. FIG. 3C shows an example of an implementation in which a flat filter 252 is placed between the lens housing 250 and the sensor 225. The achromatic doublet 240 of FIG. 3C, which incorporates a flat filter 252 with multi-band transmission characteristics, corrects optical aberrations without significantly contributing to the optical power of each lens contained in the housing 250. FIG. 3D shows another example of an implementation in which a multi-bandpass coating is implemented by applying a multi-bandpass coating 254 to the front surface of the lens assembly contained in the housing 250. Thus, this multi-bandpass coating 254 may be applied to any curved surface of any optical element contained in the housing 250.

Single Aperture Multispectral Imaging System and Method FIG. 4(a) shows a system diagram of an example multispectral imaging system according to the present disclosure. In this design, the color camera is an image sensor, although a variety of other types of image sensors may be used in accordance with the techniques of the present invention. The multispectral imaging system may include a motorized zoom lens to adjust the field of view (FOV) to capture images. Electric zoom lenses can be advantageous in that the field of view can be adjusted according to various shooting situations. For example, in biomedical applications in wound imaging, a wide field of view may be desirable for imaging large area burns, while a narrow field of view may be desirable for imaging small area features such as diabetic foot ulcers (DFUs). Field of view may be desirable. In an exemplary embodiment, the diameter of the field of view may range from 8 cm to 23 cm, but is not limited thereto. An 8-band optical filter is placed between the motorized zoom lens and the color camera, and these 8 frequency bands are designed as multispectral imaging bands. In this particular example, the center wavelengths of the eight frequency bands may be 420nm, 525nm, 581nm, 620nm, 660nm, 726nm, 820nm, and 855nm, respectively. Eight wavelengths of LED light with the same spectral band can be provided for illumination. In some embodiments, each frequency band LED can be selectively turned on and off independently of other LEDs. In front of each LED is a bandpass optical filter with a spectral band matched to each LED, which is used to confine the spectrum of illumination light from each LED.

FIG. 4b shows the quantum efficiency (Q%) of the RGB sensor of the color camera. The blue, green, and red lines represent the blue, green, and red channels of the color camera, respectively. FIG. 4c shows the transmission coefficient of the 8-band filter, which means that there are a total of eight bandpass bands in this spectrum. These eight bandpass bands are desirable as multispectral imaging bands. Similarly, FIG. 4d shows the illuminance (I ₀ ) of eight independent LEDs on the spectrum. These LED illumination bands can be overlapped with the 8-band filter bands.

5 illustrates an example of a time series of LED illumination during acquisition of multispectral imaging (MSI) and the corresponding detection coefficients for each channel. This LED illumination time series may be implemented in conjunction with the multispectral imaging system illustrated in FIG. 4. The time series illustrated in FIG. 5 is an example of an example of illumination and spectral unmixing that may be utilized within the scope of the present technology. It is understood that the multispectral imaging system illustrated in FIG. 4 may be used in conjunction with various LED time series and/or the time series illustrated in FIG. 5 may be used in conjunction with various multispectral imaging systems without departing from the spirit or scope of the present technology.

In the first step, the LED lights of channels 1, 3 and 5 (420 nm, 581 nm and 660 nm) in FIG. 5 (I) are turned on. In this time window, one image is captured by the color camera. Three wavelengths pass through the optical filter and are captured by the RGB channels of the color camera, and these three wavelengths have different filter transmission coefficients (T%) and RGB detection coefficients (Q%) as shown in (I). Next, in the second step, the LED lights of channels 1, 3 and 5 are turned off, and the LED lights of channels 2, 4 and 6 (525 nm, 620 nm and 726 nm) are turned on, as shown in (II). As above, by capturing one image by the color camera in this time window, three wavelengths with specific coefficients are captured by the three camera channels. In the third step, as shown in (III), the LED lights of channels 2 and 4 are turned off (while channel 6 of 726 nm remains on), and the LED lights of channels 7 and 8 (820 nm and 855 nm) are turned on, and a third image is obtained by the color camera for the wavelengths of channels 6, 7, and 8 with specific coefficients.

In some embodiments, an initial calibration may be performed. First, three images of a standard plate (for example, a white Zenith target) are acquired using a color camera in the following order. That is, (1) when the LED lighting is not turned on; (2) when the LED lights of channel 1, channel 3 and channel 5 (420nm, 581nm and 660nm) are turned on (other LED channels are turned off); ( 3) When channel 2, channel 4 and channel 6 (525nm, 620nm and 726nm) LED lights are turned on (other LED channels are turned off); and (4) channel 6, channel 7 and channel 8 (726nm, 820nm) At the point when the LED light of 855nm and 855nm is turned on (other LED channels are turned off), three images of the standard board are acquired with a color camera. In some embodiments, the standard can be a white Zenith target that reflects about 95% of the photons across the spectrum, and can be used to perform flat field correction of the illumination.

式中、I₀は、入射照度であり、R%は、イメージングターゲットの反射係数であり（この反射係数が、計算の対象となる未知のスペクトルに相当する）、T%は、LEDフィルタと８バンドフィルタの透過係数であり、Q%は、カラーカメラの量子係数である。カラーカメラの画像は、３つの既知量として３つのチャネル（青色、緑色、赤色）を有し、LED照明は、同時に３つの波長しか含んでおらず、３つのスペクトル帯域の反射係数（例えば、チャネル１、チャネル３およびチャネル５のR%）が、同時に計算される３つの未知量であることから、以下の行列演算式（一例として、チャネル１、チャネル３およびチャネル５を用いている）に基づくアンミキシング行列によって、３つのチャネルの未知量であるR%を求めることができる。

式中、［R%_{チャネル１} R%_{チャネル２} R%_{チャネル３}］は３つの未知量であり、［画像_青，画像_緑，画像_赤］は３つの既知量であり、［I₀×T%×Q%］は、アンミキシング係数行列である。したがって、反射係数は以下の式で求めることができる。

Next, three images of the imaging target are acquired with the color camera in the same order (e.g., no LED illumination; then channel 1, channel 3, and channel 5; then channel 2, channel 4, and channel 6; and finally channel 6, channel 7, and channel 8). By performing background subtraction and flat-field correction, the following formula can be used:

In the formula, _I0 is the incident illuminance, R% is the reflection coefficient of the imaging target (this reflection coefficient corresponds to the unknown spectrum to be calculated), T% is the transmission coefficient of the LED filter and the 8-band filter, and Q% is the quantum coefficient of the color camera. Since the image of the color camera has three channels (blue, green, and red) as three known quantities, and the LED illumination only contains three wavelengths at the same time, the reflection coefficients of the three spectral bands (e.g., R% of channel 1, channel 3, and channel 5) are the three unknown quantities to be calculated at the same time, the unknown quantity R% of the three channels can be obtained by the unmixing matrix based on the following matrix operation formula (using channel 1, channel 3, and channel 5 as an example).

In the formula, [R% _{channel 1,} R% _{channel 2,} R% _{channel 3} ] are three unknowns, [image _blue , image _green , image _red ] are three knowns, and [I ₀ ×T%×Q%] is the unmixing coefficient matrix. Therefore, the reflection coefficient can be calculated by the following formula.

Similarly, the reflection coefficients of other channels, namely, channel 2, channel 4, channel 6, channel 7 and channel 8 (525 nm, 620 nm, 726 nm, 820 nm and 855 nm), when LED lighting is turned on with a specific wavelength combination, are also obtained by the corresponding unmixing matrices. Therefore, eight MSI images are generated based on the images for each channel obtained by the three unmixing matrices.

The combination of image acquisition time series and LED illumination described above with reference to FIG. 5 is an example of a method for acquiring eight MSI images. In practice, the time series of LED illumination and image acquisition can be designed in various combinations and/or at various timings, and the combination of the time series of LED illumination and image acquisition depends on hardware performance, heuristics, etc. It may be selected and used based on the following. Alternatively, a monochrome camera can be used if each LED wavelength is designed to be lit individually, so no unmixing matrix is required. In this case, MSI performance can be evaluated by photographing the target and comparing the quality of the MSI image with the true value of the spectrum (i.e. color calibration).

Illustrative Single Aperture Multispectral Imaging System Implementations and Results FIGS. 6 and 7 show an illustrative embodiment of a single aperture , single camera, multispectral image capture device that may be implemented using the MSI system and methods disclosed herein. FIG. 6 is a perspective view of a handheld device with an LED array and a camera for performing the multiplexed illumination described above. Included within the handheld device shown in FIG. 6 may be one or more processors and memory for storing captured image data. Further image data processing, such as spectral unmixing as disclosed herein and/or generation of individual spectral images as described below in this specification, may be performed within the handheld device shown in FIG. 6 or may be performed solely under the full or partial control of one or more remote computing devices or in conjunction with the handheld device shown in FIG. 6 under the full or partial control of one or more remote computing devices without departing from the scope of the present disclosure.

FIG. 7 shows an example of an LED configuration including a total of 24 LEDs. In this LED configuration, three LEDs of eight types with different wavelength bands are arranged in a ring to surround a single aperture of the image acquisition device. To reduce unmixing errors due to complexity, the number of colors in LED lighting is increased to eight wavelengths. Individual LEDs may cover predetermined MSI spectral bands, with center wavelengths of about 420 nm, about 525 nm, about 581 nm, about 620 nm, about 660 nm, about 726 nm, about 820 nm, and about 880 nm, respectively. Since this image sensor has a single aperture design, a new spatial arrangement of the LEDs was developed to reduce the size of the front of the lighting board and image sensor. The annular design provides desirable photon collection efficiency from the LED illumination to the central camera aperture. To obtain uniform illumination at a desired working distance (e.g., 40 cm in some embodiments), each LED color on the lighting board may be composed of up to three or more LEDs, with each LED The beam aperture may be 120°. A diffuser plate can be placed on the lighting board. Place spacers in place to ensure sufficient distance between the LED and the diffuser to prevent heat damage to the diffuser. Using one or more processors integrated into the imaging head (e.g., the handheld device shown in Figure 6), individual LEDs can be controlled. Furthermore, with such a configuration, a high signal/noise ratio can be obtained by adjusting the calibration time of the illuminance of each light source.

8-10 show the results of tests with a single aperture system according to the techniques of the present invention. Figure 8 shows the RGB images of the Macbeth and Zenith targets taken using a single lens, single aperture, and single camera system under conditions of illumination with seven types of color LEDs and subtracted against the background of the surrounding indoor lights. show. In this test example, seven types of color LEDs (blue, green, PC amber, deep red, far infrared, NIR I and NIR II) were placed on the lighting board and turned on and off individually in chronological order. Eight types of bandpass optical filters for MSI (center wavelength/full width at half maximum (FWHM): 420/10nm, 520/10nm, 580/10nm, 620/10nm, 660/10nm, 730/10nm, 810/10nm and 850/10nm ), a total of 7 × 8 = 56 images were acquired on the RGB camera.

測定の目的はターゲットの反射率（R）であり、検出されたシグナル（S）が直接的な測定値である。８バンドフィルタは、８つの所定のMSIスペクトルウィンドウのみに光の透過を閉じ込めることから、Rの未知の測定値は８つのみである。７種のカラーLED照明（I）を調節し、量子効率（Q）が同じ３つのスペクトルセンサ（すなわちRGBチャネル）を用いることによって、十分な既知量（7×3=21）を用いたリニアなアンミキシング行列演算を行って、８つの未知量を求めることができる。このプロセスにより生成された８枚のMSI画像を図９に示す。 A linear post-processing method summed eight images captured with eight filters using the same LED color, which was equivalent to seven RGB images captured under seven illumination conditions filtered through eight band filters. A Macbeth target was imaged for spectroscopic analysis, and a Zenith target with 95% reflectance was imaged for flat-field correction. Eight MSI images were then generated using an unmixing algorithm. The principles of unmixing are outlined herein. The detected signal (S) is the integral of the illuminance (I) multiplied by the reflectance of the target (R), the transmission coefficient (T) of the optical system including the lens and bandpass filter, and the quantum efficiency (Q) of the sensor in the spectral band (λ), as follows:

The measurement objective is the reflectance (R) of the target, and the detected signal (S) is the direct measurement. The 8-band filter confines the transmission of light only to the 8 predefined MSI spectral windows, so there are only 8 unknown measurements of R. By tuning the 7 color LED illumination (I) and using 3 spectral sensors (i.e., RGB channels) with the same quantum efficiency (Q), a linear unmixing matrix operation with enough known quantities (7 x 3 = 21) can be performed to determine the 8 unknown quantities. The 8 MSI images generated by this process are shown in Figure 9.

After generating the MSI images, we analyzed the spectral accuracy by comparing the MSI spectra of each patch of the Macbeth target with the true values measured with a calibrated spectrometer. The comparison results are shown in Figure 10. The comparison results showed that the improved single-aperture MSI system design proposed by the present invention has higher accuracy of spectral measurements (average correlation coefficient = 0.97).

Terminology All methods and tasks described herein may be performed by computer systems or may be fully automated. In some cases, the computer system may include multiple separate computers or computing devices (e.g., physical servers, workstations, storage arrays, cloud computing devices) that communicate and interoperate over a network to perform the functions described herein. resources, etc.). Any such computing device typically includes programs stored in memory or other non-transitory computer-readable storage media (e.g., solid-state storage devices, disk drives, etc.). Includes a processor (or processors) that execute instructions or program modules. The various functions disclosed herein may be embodied in such program instructions or implemented in the form of special purpose circuitry (eg, an ASIC or FPGA) of a computer system. When a computer system includes multiple computing devices, these devices may be co-located or located at separate locations. The results of the methods and tasks disclosed herein may be permanently stored by converting physical storage devices, such as solid-state memory chips or magnetic disks, into various forms. In some embodiments, the computer system may be a cloud-based computing system where processing resources are shared by multiple separate business entities or other users.

The processes disclosed herein may be initiated in response to an event, such as a predefined schedule, a dynamically determined schedule, or some other event, upon request initiated by a user or system administrator. Once the processes are initiated, executable program instructions stored on one or more non-transitory computer-readable media (e.g., hard drives, flash memory, removable media, etc.) may be loaded into memory (e.g., RAM) of a server or other computing device. The executable instructions may then be executed by a hardware-based computer processor included in the computing device. In some embodiments, the processes disclosed herein, or portions thereof, may be implemented on multiple computing devices and/or multiple processors, either serially or in parallel.

Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein may be performed in a different order, may be added to one another, may be combined with one another, or may be omitted altogether (e.g., not all of the acts and events described herein are required to perform the algorithms described herein). Furthermore, in certain embodiments, multiple acts or events may be performed simultaneously rather than sequentially, for example, using multithreaded processing, interrupt processing, or multiple processors or multiple processor cores, or on other parallel structures.

The various exemplary logic blocks, modules, routines, and algorithm steps described in connection with the embodiments disclosed herein may be implemented in a computer operating on electronic equipment (e.g., an ASIC or FPGA device), computer hardware. It can be implemented as software or a combination of these. Additionally, the various exemplary logic blocks and modules described in connection with the embodiments disclosed herein may include processor devices, digital signal processors ("DSPs"), application specific integrated circuits ("ASICs"), Field programmable gate arrays (“FPGAs”) or other programmable logic devices, discrete gate logic or transfer logic, discrete hardware components, or combinations of these components designed to perform the functions described herein It can be implemented with equipment such as. The processor device may be a microprocessor, but in other aspects the processor device may be a controller, a microcontroller or a state machine, combinations thereof, and the like. A processor device may include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor device includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. The processor device may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of a DSP core and one or more microprocessors, or such other combinations. It can be implemented as a configuration. Although described herein primarily with respect to digital technology, the processor device may include primarily analog components. For example, all or some of the rendering techniques described herein may be implemented using analog circuitry or a combination of analog and digital circuitry. The computing environment may include any type of computer system, including a microprocessor-based computer system, a mainframe computer, a digital signal processor, a portable computing device, a device controller, to name a few. , computer engines in electrical appliances, etc., but are not limited to these.

Components of the methods, processes, routines or algorithms described with respect to the embodiments disclosed herein may be directly embodied in hardware or in software modules executed by a processor device. , or can be directly realized by a combination of these hardware and software modules. Software modules may be contained in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or other forms of non-transitory computer-readable storage media. . Typical storage media may be coupled to a processor unit such that the processor unit can read information from, and write information to, such storage media. In this aspect, the storage medium may be integral to the processor device. The processor unit and storage medium may be included in an ASIC. This ASIC may be included in the user terminal. In this aspect, the processor device and storage medium may exist as discrete components of a user terminal.

As used herein, terms indicating conditions, such as "may," "may," "may," "may," and "for example," are used unless expressly stated otherwise. Typically, a particular embodiment includes certain features, components, or steps, and another embodiment does not include those features, unless the context in which those terms are used indicates otherwise. , means that no component or step is included. Accordingly, such terminology generally refers to the presence or absence of other input or prompting, regardless of whether a particular feature, component, or step is included or performed in a particular embodiment. does not imply that one or more embodiments require these features, components or steps in any way or that one or more embodiments necessarily include the logic of a decision in the absence of one or more embodiments; do not. Additionally, terms such as "comprising," "comprising," "having," and the like are synonymous and are used in an open-ended, inclusive sense and do not exclude additional components, features, acts, operation, etc. do not have. Additionally, the term "or" is also used in an inclusive (rather than an exclusive) sense, e.g., when the term "or" is used to list constituents, the listed constituents Refers to one, part or all of the elements.

Unless otherwise specified, disjunctive terms such as "at least one of It is understood that it is used to indicate that any combination of these (eg, X, Y or Z) may be present. Thus, such disjunctive terms are generally understood to mean that the presence of each of at least one of X, at least one of Y, and at least one of Z is required in a particular embodiment. Unintentionally, and it should not be intended that way.

While the foregoing detailed description has shown, described, and pointed out novel features as applied to various embodiments, the form of the foregoing apparatus or algorithm and the details thereof may be omitted without departing from the scope of this disclosure. , it will be understood that various omissions, substitutions and modifications may be made. Certain embodiments described herein may incorporate all of the features and advantages described herein, since some of the features described herein can be used or implemented separately from other features. It will be easily understood that the present invention may be embodied in a form other than that provided. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims (30)

1. A multispectral imaging system, comprising:
a light source configured to selectively emit light comprising one or more of a set of four or more predetermined frequency bands to illuminate an object;
an image sensor configured to receive a portion of the emitted light that is reflected by the object;
an aperture positioned to pass a portion of the emitted light that is reflected by the object and incident on the image sensor;
a multi-bandpass filter disposed over the aperture and configured to pass light in the four or more predetermined frequency bands;
a memory having instructions stored thereon for generating a multispectral image and performing unmixing; and at least one processor,
The at least one processor performs at least the following steps in accordance with the instructions:
emitting light from the light source in a first subset of two or more of the predetermined frequency bands;
receiving first image data from the image sensor generated based on the reflected light of a first subset of light;
emitting light from the light source in a second subset of two or more of the predetermined frequency bands;
receiving second image data from the image sensor generated based on the reflected light of a second subset of light;
processing the first image data and the second image data to generate at least a first multispectral image and a second multispectral image;
configured to perform spectral unmixing to generate a plurality of images of the object in a single frequency band, each image corresponding to one of the four or more predetermined frequency bands.
Multispectral imaging system. The processor,
emitting light from the light source in a third or more subset of two or more frequency bands of the predetermined frequency bands; and based on reflected light of the third or more subset of frequency bands; 2. The multispectral imaging system of claim 1, further configured to receive third or more image data generated by the image sensor from the image sensor. The multispectral imaging system of claim 1 or 2, wherein the light source includes a plurality of light emitting diodes (LEDs), each LED being configured to emit light in any one of the four or more predetermined frequency bands. 4. The multispectral image of claim 3, wherein the at least one processor is further configured to select frequency bands simultaneously emitted by the light source by controlling activation of individual LEDs of the light source. system. 5. The multispectral imaging system of claim 4, wherein the processor controls activation of individual LEDs by controlling an electronically switched shutter that selectively passes or blocks light emitted by each LED. Claims 1 to 3, wherein the multi-band pass filter comprises a plurality of frequency bands for passing light, each frequency band corresponding to any one of the four or more predetermined frequency bands. 5. The multispectral imaging system according to any one of 5. The multispectral imaging system of any one of claims 3 to 6, wherein the LEDs and the multi-bandpass filter have the same number of frequency bands, and each frequency band of the LEDs is aligned to match a corresponding frequency band of the multi-bandpass filter. 7. According to any one of claims 3 to 6, further comprising a bandpass filter placed on each individual LED of the plurality of LEDs, whereby the light emitted by each individual LED is confined to a precise spectrum. multispectral imaging system. The multispectral imaging system of claim 3, wherein the set of four or more predetermined frequency bands includes eight predetermined frequency bands, and the light source includes eight LEDs, each LED configured to emit light in one of the predetermined frequency bands. The multispectral imaging system of claim 9, wherein the multi-bandpass filter includes an eight-band filter configured to pass each of the eight predetermined frequency bands. The multispectral imaging system of claim 10, wherein the at least one processor is further configured to, in accordance with the instructions, cause the light source to emit light of a third subset of the predetermined frequency band, and to receive third image data from the image sensor that is generated based on reflected light of the third subset of light. 12. The multispectral imaging system of claim 11, wherein each of the first, second and third subsets includes three frequency bands of the predetermined frequency bands. 10. The multispectral imaging system of claim 9, wherein the predetermined frequency band is defined by a center wavelength ranging from ultraviolet (UV) to short wavelength infrared. 14. The multispectral imaging system of claim 13, wherein the center wavelengths of the predetermined frequency bands are 420nm, 525nm, 581nm, 620nm, 660nm, 726nm, 820nm, 855nm, respectively. The multispectral imaging system of any one of claims 1 to 14, further comprising a motorized parfocal zoom lens or a motorized varifocal zoom lens disposed over the aperture, and the one or more processors are further configured to control the motorized zoom lens to adjust a field of view (FOV) of the multispectral imaging system. The multispectral imaging system of claim 15, wherein the parfocal zoom lens or the variable focus zoom lens is configured to allow automatic or manual focus adjustment by changing the focal length (FL) of the lens. The multispectral imaging system of claim 15 or 16, wherein adjusting the field of view of the multispectral imaging system does not change the imaging distance between the object and the image sensor. The multispectral image sensor of any one of claims 15 to 17, wherein the one or more processors are further configured to adjust the field of view of the multispectral imaging system by changing a shooting distance between the object and the image sensor. The multispectral image sensor of claim 18, wherein the processor is further configured to compensate for contrast loss caused by the zoom lens. The multispectral imaging system of any one of claims 15 to 19, wherein the field of view is adjustable over a range of at least 8 cm to 23 cm. 21. The multispectral imaging system of claim 20, wherein the multispectral imaging system retains high spatial resolution without reducing the number of samples in the field of view, unlike post-processing digital cropping methods that sacrifice resolution. Spectral imaging system. The multispectral imaging system of any one of claims 1 to 21, wherein the one or more processors are further configured to visualize the object in natural pseudocolor based on the multispectral image. 23. The multiprocessor according to any one of claims 1 to 22, wherein the one or more processors are configured to perform spectral unmixing by determining the reflection coefficient of the object using matrix arithmetic expressions. Spectral imaging system. 24. The multispectral imaging system of claim 23, wherein the reflection coefficients are determined based at least in part on the values of incident illuminance, the values of the transmission coefficients of the multiple bandpass filters, and the values of the quantum coefficients of the image sensor for each channel. Multispectral imaging system according to any one of claims 1 to 24, capable of taking three or more multispectral images in less than 100 milliseconds. The multispectral imaging system of any one of claims 1 to 25, wherein the object is a tissue region. 27. The multispectral imaging system of claim 26, wherein the tissue includes a wound, cancer, ulcer, or burn. 28. The multispectral imaging system of claim 27, wherein the wound comprises a diabetic ulcer, a non-diabetic ulcer, a chronic ulcer, a post-operative wound, an amputation site, a burn, a cancerous lesion, or damaged tissue. 27. The multispectral imaging system of claim 26 for use in identifying a tissue classification or in identifying a tissue healing score, wherein the tissue classification is, for example, living or healthy tissue, dead tissue or necrotic tissue, perfused or non-perfused tissue, or ischemic or non-ischemic tissue, the healing score of which is at least 50% of the wound, e.g. after 30 days of standard wound care treatment. A multispectral imaging system that has a tendency to heal or at least 50% of the wound not to heal after 30 days of standard wound care treatment. 27. The multispectral imaging system of claim 26 for use in imaging a wound, cancer, ulcer or burn, wherein the wound, cancer, ulcer or burn is, for example, a diabetic ulcer, a non-diabetic ulcer, a chronic ulcer, a post-operative wound, an amputation site, a burn, a cancerous lesion or damaged tissue.