WO2021262129A1

WO2021262129A1 - An artificial intelligence analysis based on hyperspectral imaging for a quick determination of the health conditions of newborn premature babies without any contact

Info

Publication number: WO2021262129A1
Application number: PCT/TR2021/050658
Authority: WO
Inventors: Murat CEYLAN
Original assignee: Ceylan Murat
Priority date: 2020-06-26
Filing date: 2021-06-25
Publication date: 2021-12-30
Also published as: TR202010104A2

Abstract

The invention is an analysis of hyperspectral images taken from babies with artificial intelligence to determine the health conditions of neonates and premature babies to be used in neonatal intensive care units.

Description

AN ARTIFICIAL INTELLIGENCE ANALYSIS BASED ON HYPERSPECTRAL IMAGING FOR A QUICK DETERMINATION OF THE HEALTH CONDITIONS OF NEWBORN PREMATURE BABIES WITHOUT ANY CONTACT

TECHNICAL FIELD

BACKGROUND

Before explaining the components of hyperspectral imaging, other imaging methods should be briefly described and given comparatively. The electromagnetic spectrum diagram in which the imaging ranges are defined is shown in Table 1.

Table-1 Electromagnetic spectrum diagram in which the imaging ranges are defined

Visible imaging; It is an imaging technique performed with devices frequently used in daily life such as mobile phones, tablets, computers, and cameras. Its electromagnetic spectrum is in the wavelength range of about 380 - 780 nm. The human eye also performs vision in this spectral range.

Infrared imaging is divided into three main regions: near, mid, and far. Near- infrared is about 780 - 5000 nm, mid-infrared is about 5000 - 25000 nm, far-infrared is about 25000 - 20000 nm. Infrared thermal imaging is located in the mid-infrared region and provides thermal maps with temperature values of the imaged surface at wavelengths of approximately 7500 - 14000 nm.

Hyperspectral imaging (HSI) is an imaging technique that enables the acquisition of spectral bands of the visible, near-infrared, mid-infrared and thermal infrared regions of the electromagnetic spectrum in large numbers and with small intervals. Its electromagnetic spectrum is in the wavelength range of ~200-15000 nm. The most important aspect that distinguishes this imaging technique from others is that it can obtain many spectral bands with narrow intervals within the imaging range. This is three spectral bands (red, green, and blue) in the range of ~380-780 nm in visible imaging and one spectral band (thermal map) in thermal imaging, while in hyperspectral imaging, it is in the range of ~1000 spectral bands in the range of ~200 - 15000 nm. Thus, hundreds of spectral bands of different wavelengths belonging to the same imaging surface are obtained. Images obtained with hyperspectral imaging are called hypercubes.

After obtaining the hypercube, analyses are performed using a pixel's spectral values under different wavelengths or by using the spectral values of all pixels at one wavelength.

BRIEF DESCRIPTION OF THE INVENTION

The invention, together with the advantages of non-contact, non-ionizing, non harmful radiation-free, non-invasive methods such as both thermal and visible imaging for use in neonatal intensive care units, together with the advantages of obtaining a large number of bands in a wider spectrum; it is about a method that quickly and intelligently diagnoses the health conditions of neonate babies by combining hyperspectral imaging with artificial intelligence, which eliminates the disadvantage of current systems to analyze limited bands.

LIST OF FIGURES

Figure 1a. Convective (closed) Type Incubator, which cannot be shown with a drawing,

Figure 1b. Open Radiant Heater (open) Type Incubator, which cannot be shown with a drawing,

Figure 2a. 397.32 nm Spectrum Image Figure 2b. 542.68 nm Spectrum Image Figure 2c. 542.68 nm Spectrum Image Figure 2d. 542.68 nm Spectrum Image Figure 3a. Spectra corresponding to All Pixels in ROIs Figure 3b. 3D ROIs from Neonate

Figure 4. 3D Data to be obtained as a result of PCA

Figure 5. 3D-CNN Model Structure

Figure 6. Single Flypercube formed by combining Neonatal Hypercubes

Figure 7. The new hypercube obtained as a result of PCA

Figure 8a. Mini-cube for Neighborhood Extraction Method

Figure 8b. Mini-cubes Created to scan All Pixels for Neighborhood Extraction Method

Figure 9. 3D-CNN Model Structure used for Neighborhood Extraction Method

DETAILED DESCRIPTION OF THE INVENTION

The invention will record hyperspectral images of babies in the neonatal intensive care unit by the camera operating in the wavelength range of 400 nm - 1000 nm (or wider). Hyperspectral images can be transferred wirelessly to a portable computer, or they can be transferred after they are saved to the SD card on it.

In our invention, babies do not need to be removed from their controlled environment, namely incubators, before hyperspectral images are recorded. According to their disease conditions, some babies are screened in open incubators and some babies in closed incubators (Figures 1a and 1b). The acquisition time required to create hyperspectral bands is expected to vary between 30 seconds and 90 seconds, depending on the light intensity. Before the measurement, spectral band calibration will be performed by the white color. The measurement will be taken at a distance of 60 cm - 100 cm from the neonate lying in the supine position. The camera angle should be about 90°. Images in different spectra are shown in Figures 2a, 2b, 2c, and 2d.

In image analysis, the size of the data used poses a problem for computer hardware and the performance of image processing and artificial intelligence algorithms. Here, the situation expressed by the size of the data is the use of many features that have the same meaning. Instead of this situation, different features representing data should be found and used in both image processing and artificial intelligence algorithms.

Hypercubes in MxNxD size were obtained from unhealthy and healthy neonate babies using a hyperspectral camera. Here M is the width, N is the height, and D is the number of spectral bands. Then, using these hypercubes, region of interest (ROI) was selected from the neonates' body as SxSxD. In other words, as seen in Figure 3b, SxSxD size data were obtained for each neonate. SxS indicates the selected ROI region, and D indicates the number of spectral bands.

Then, due to the large size of the data, in order to avoid the computational cost, the dimensions of the data were reduced by applying the Principal Component Analysis (PCA), which is one of the dimension reduction methods, to all 3D data separately so that the spatial dimension does not change. As a result of this process, new hypercubes of 5x5x2. size were obtained (Figure 4).

As seen in Figure 5 in the artificial intelligence software part of the invention, 3D convolutional neural networks architecture (3D-CNN) 3 convolution (Conv3d), 2 pooling (MaxPool3d), 1 flattening (Flatten) and 4 fully connected (FC) network layers was created using In addition, overfitting is prevented by using the Dropout layer after each FC layer. As shown in Figure 5, important feature maps were obtained by using three-dimensional convolution layers and obtaining spatial and spectral features.

Pixel-oriented classification with Neighborhood Extraction using 3D-CNN

The sizes of the regions of interest selected from the neonates were determined as MxNxD. Finally, ROIs from different neonates in total were combined to form a single hypercube. The size of this created hypercube is 2 SxSxD and is shown in Figure 6. In this way, the black areas show unhealthy neonates, while the white ones show healthy neonates. The purpose of obtaining a single hypercube is to apply the Neighborhood Extraction method.

The PCA method was applied to this hypercube. The spatial dimension would not change, and the spectral dimension was reduced to l. The newly formed hypercube is shown in Figure 7.

In order to create the labels, 0 labels were assigned to all the pixels obtained from unhealthy neonates in the black region. In the same way, real labels were created by assigning 1 label value to all pixels in the white region. That is, a total of E_hasta

2-label values were created, with 5 * (5/2) — £ i_asta labels being 0 and 5 * (5/2) = E_sa^_hkh label values being 1. Then, 3D mini-cubes (3D neighboring patches) with the size of WxWxX (Figure 8a) were created separately, with these pixels as the center point. Label values at the center point were assigned to these cubes. These mini cubes are shown in detail in Figure 8b. Since all pixels will be included here, first of all, the frame is expanded in k size by performing the zero-padding operation. The k value is calculated as follows: k = (W - l)/2

The 3D-CNN model used for the Neighborhood Extraction method is given in Figure 9.

In the light of the technical information given above, the hyperspectral imaging and analysis method for use in neonatal intensive care units, which is the subject of our invention, is carried out with the following stages:

- Obtaining hyperspectral images from premature neonate babies lying in the incubator using a hyperspectral camera,

- Separation of the hyperspectral bands of the acquired hyperspectral images,

- Extracting the spectral signatures of premature babies from hyperspectral bands and determining the difference between the spectral signatures of babies diagnosed with the disease and healthy babies,

- Analyzing hyperspectral bands, hyperspectral signatures, and hyperspectral images with artificial intelligence software,

- Classification of hypercubes obtained from hyperspectral data of premature babies with a three-dimensional convolutional neural network (3D-CNN) using the neighborhood extraction method developed specifically for this problem, and as a result of this classification, premature babies are automatically (without the need for specialist intervention) diagnosed as unhealthy and healthy.

Claims

1 . An artificial intelligence analysis method based on hyperspectral imaging for the non-contact and quick determination of the health status of neonate premature babies comprising the steps below;

- Separation of the hyperspectral bands of the acquired hyperspectral images,