TWI796278B - Chemo-brain image visualization classifying system and operating method thereof - Google Patents

Abstract

This invention discloses a chemo-brain image visualization classifying system and operating method thereof. The chemo-brain image visualization classifying system firstly executes a pre-processing program, therefore to acquire a post-processed whole brain MRI image from an original whole brain MRI image. Thereinafter, this invention extracts the mean fractional amplitude of low-frequency fluctuations (mfALFF) data from the post-processed whole brain MRI image and inserts the mean fractional amplitude of low-frequency fluctuations (mfALFF) data into a well-trained volumetric convolutional neural network model. At last, the well-trained volumetric convolutional neural network model predicts a whole brain predicted result, and an integrated gradients algorithm converts the whole brain predicted result into a visualized whole brain cerebral region image.

Description

Chemotherapy Brain Imaging Visual Classification System and Its Operation Method

The present invention is a chemotherapy brain image visualization classification system and its operation method, in particular, a trained Volume Convolutional Neural Network (Volumetric Convolutional Neural Network) model and integrated gradients (integrated gradients) algorithm can be used simultaneously to convert Magnetic resonance imaging (Magnetic resonance imaging, MRI) images are predicted, classified and visualized based on the predicted results, and the chemotherapy brain image visualization classification system and its operation method.

In modern society, cancer has become the biggest killer of human beings in their later years. Among them, breast cancer is the most common female cancer in the world. As with all breast cancer treatment modalities, breast cancer survivors who have undergone chemotherapy may experience cognitive impairment during or after cancer treatment. Generally speaking, this phenomenon is called chemical brain (chemo-brain).

The so-called chemo-brain is often used to describe changes in cognitive function after systemic chemotherapy. Due to the influence of the chemical brain (chemo-brain), most survivors with this phenomenon cannot accurately describe their condition. For example, traditional doctors and nurses will use cognitive assessment scales to conduct tests and communicate with patients; however, for patients with a chemo-brain, it may be unavoidable for patients to give false answers. It will further affect the follow-up treatment and follow-up at the first time.

In addition, in view of the fact that the chemical brain (chemo-brain) is actually very difficult to identify directly from the magnetic resonance imaging (MRI) images with the naked eye; therefore, an objective interpretation system is urgently needed to assist in judging survivors Whether it has the possibility of chemical brain (chemo-brain).

In order to solve the problems mentioned in the prior art, the present invention provides a visual classification system for chemotherapy brain images. The chemotherapy brain image visualization classification system is used to run the operation method of the chemotherapy brain image visualization classification system described later. Specifically, the image visual classification system includes a processing module and at least one storage module. Wherein, the at least one storage module is connected with the processing module.

Further, the operating method of the visualization classification system for chemotherapy brain images mainly consists of the following steps. Firstly, step (A) is to provide at least one original whole brain MRI image to a processing module. Next, in step (B), the processing module executes an image pre-processing means on the at least one original whole-brain MRI image to obtain at least one pre-processed whole-brain MRI image.

Next, the step (C) is to extract at least one mean fractional amplitude of low-frequency fluctuations (Mean fractional amplitude of low-frequency fluctuations, mfALFF) of the whole brain from the at least one pre-processed whole brain MRI image by the processing module. Information.

In step (D), the processing module inputs the at least one whole-brain average low-frequency fluctuation amplitude fraction information into a volumetric convolutional neural network (Volumetric Convolutional Neural Network) model for prediction, and obtains at least one whole-brain prediction result.

Finally, in step (E), the processing module converts the at least one whole-brain prediction result into at least one whole-brain region image through an integrated gradients algorithm. Among them, the volumetric convolutional neural network model includes 3D-ResNet-50 deep learning model and 3D-DenseNet-121 deep learning model.

The purpose of the above brief description of the present invention is to make a basic description of several aspects and technical features of the present invention. The summary of the invention is not a detailed description of the invention, so it is not intended to specifically list the key or important elements of the invention, nor is it used to define the scope of the invention, but to present several concepts of the invention in a concise manner.

In order to understand the technical features and practical effects of the present invention, and to implement it according to the contents of the specification, the preferred embodiment shown in the drawings will be described in detail as follows:

First, please refer to FIG. 1. FIG. 1 is a system architecture diagram of an embodiment of the visualization classification system for chemotherapy brain images of the present invention. As shown in FIG. 1 , the visualized classification system 10 for chemotherapy brain images in this embodiment is mainly composed of a processing module 100 and a storage module 200 a and a storage module 200 b connected to the processing module 100 .

Further, the processing module 100 of this embodiment may be a central processing unit (Central Processing Unit, CPU), a microprocessor (Microprocessor), a graphics processing unit (Graphics Processing Unit, GPU), a specific application integrated circuit ( Application Specific Integrated Circuit, ASIC) or a combination thereof. The storage module 200a or 200b can be random access memory (RAM), read only memory (ROM), flash memory (flash memory), hard disk (hard disk), floppy disk (soft disk), database (database) or a combination thereof. The invention is not limited.

In this embodiment, the storage module 200a can preferably be implemented as random access memory (RAM), read only memory (ROM), flash memory (flash memory), hard disk (hard disk) or Hardware with memory function such as floppy disk (soft disk). In contrast, the storage module 200b can be a database (database) such as a cloud.

In this embodiment, the storage module 200a and the storage module 200b can be used, but not limited to, to store all the software, algorithms, programs, data sets (Data set), etc. that the processing module 100 needs to run or use. Not limited.

Next, please refer to FIG. 2 . FIG. 2 is a flow chart of an embodiment of the operating method of the chemotherapy brain image visualization classification system of the present invention. Specifically, the operation method of the visualization classification system for chemotherapy brain images shown in FIG. 2 runs under the environment framework of the visualization classification system 10 for chemotherapy brain images shown in FIG. 1 .

According to this embodiment, before implementing the operation method of the chemotherapy brain image visualization classification system, the volumetric convolutional neural network (Volumetric Convolutional Neural Network) model applicable to this embodiment must be trained.

Specifically, the volumetric convolutional neural network model in this embodiment includes a 3D-ResNet-50 deep learning model and a 3D-DenseNet-121 deep learning model. Among them, the 3D-ResNet-50 deep learning model is formed by embedding the compression-excitation block (SE-block) into the ResNet-50 deep learning model through the residual connection (Skip Connection). The 3D-DenseNet-121 deep learning model is formed by embedding the compression-excitation block (SE-block) into the dense block of the DenseNet-121 deep learning model. Moreover, the 3D-ResNet-50 deep learning model embeds the compression-excitation block (SE-block) embedded in the dense block (dense block) of the DenseNet-121 deep learning model through the residual connection (Skip Connection). Accordingly, the 3D-ResNet-50 deep learning model and the 3D-DenseNet-121 deep learning model of this embodiment are closely connected.

Next, in this embodiment, the information of 55 breast cancer survivors after chemotherapy and 65 healthy control groups was collected to train the volume convolutional neural network model of this embodiment. First of all, the calculation requirements for image processing in this embodiment are all realized through the visualization and classification system 10 for chemotherapy brain images, which will be described first.

Specifically, in this embodiment, a total of 120 subjects obtained original whole-brain MRI images by means of functional Magnetic Resonance Imaging (fMRI). In this stage, the scanning parameters of functional Magnetic Resonance Imaging (fMRI) were set as TR/TE = 2000/30 milliseconds, flip angle (flip angle) was 90 ^◦ , NEX = 1, FOV = 220 × 220 mm ² , matrix size = 64 × 64 and voxel size = 3.4 × 3.4 × 4 mm ³ . Finally, 31 axial images will be extracted to cover the whole brain. Each resting-state fMRI run contained 300 image volumes. The overall scan time is about 10 minutes.

Next, the obtained original whole-brain MRI images will be processed by means of image pre-processing into a state that can be input and used to train the volume convolutional neural network model of this embodiment. Specifically, the original whole-brain MRI images obtained in the previous stage were processed with Statistical Parametric Mapping (SPM). In this embodiment, the Statistical Parametric Mapping (SPM) is implemented by software SPM12 (Wellcome Department of Cognitive Neurology, London, UK). The steps performed include slice timing correction (Slice Timing) correction step, displacement correction (Realignment) step, normalization (Normalization) step, spatial smoothing (Smoothing) step or a combination thereof, only according to the actual situation of the original whole brain MRI image. Condition adjustment is not limited by the present invention.

Specifically, in this embodiment, after performing the image pre-processing means, the unqualified at least one original whole-brain MRI image will be excluded through a screening criterion. The screening criteria are to exclude the at least one original whole-brain magnetic field whose head motion (Translation) distance exceeds 1 millimeter (mm) or head motion (Head motion) rotation (Rotation) angle exceeds 1 degree Vibration imaging.

And when the image pre-processing means of this embodiment executes the normalization (Normalization) step, the normalization (Normalization) step is the at least one original whole-brain MRI image after the displacement correction (Realignment) step in the MNI space ( Standard Montreal Neurological Institute space) was used as the sample back of the brain, and was standardized with 3 mm isotropic voxels (isotropic 3 mm voxels).

Further, when the image pre-processing means of this embodiment executes the spatial smoothing (Smoothing) step, the spatial smoothing (Smoothing) step is 6 millimeters (mm) full width at half maximum (Full width at half maximum, FWHM ) Gaussian filter (Gaussian filter) to amplify the signal-to-noise ratio of the at least one original whole-brain MRI image.

Finally, for the pre-processed whole brain MRI image processed by the image pre-processing means, this embodiment will further filter the pre-processed image with a frequency between 0.01-0.12 Hz through a band pass filter. Magnetic resonance imaging of the whole brain. Accordingly, the mean fractional amplitude of low-frequency fluctuations (mfALFF) information of the whole brain that can be used for training is extracted. In this embodiment, the software whose filtering frequency is between 0.01-0.12 hertz (Hz) can be realized by using the Resting-State Data Analysis Toolkit (version 1.8), which is not limited by the present invention.

Since the total number of samples used for training in this embodiment is 120, a total of 120 whole brain average low-frequency fluctuations (Mean fractional amplitude of low-frequency fluctuations, mfALFF) information will be generated for training The volumetric convolutional neural network model of the embodiment. In the training process of this embodiment, the growth rate of the volumetric convolutional neural network model is 16 through the framework of Tensoflow and Keras higher API. Next, the whole brain mean fractional amplitude of low-frequency fluctuations (mfALFF) information from 5 breast cancer survivors after chemotherapy and the whole brain mean fractional amplitude of low-frequency fluctuations (mfALFF) from 5 healthy controls (Mean fractional amplitude of low-frequency fluctuations, mfALFF) information will be used as the test set (test set); ) information will be used for training, and its training results will be verified through 10-fold cross validation.

In this embodiment, the input size (input size) of all image samples input into the volumetric convolutional neural network model (that is, the mean fraction amplitude of low-frequency fluctuations (mfALFF) information of the whole brain) ) are 64 × 64 × 64 × 1. All image samples exceeding the input size of the volumetric convolutional neural network model are predefined by a grid-search method.

Further, the volumetric convolutional neural network model of this embodiment is trained using an SGD optimizer (SGD optimizer), and the learning rate (Learning Rate) is set to 1e-2. And the batch size (batch size) is set to 5. The L2 regression factor (regularization factor) is predefined as 1e-4. In addition, in this embodiment, a random rotation data method is used to amplify the data to avoid overfitting of image samples. Finally, in this embodiment, the training epoch of the volumetric convolutional neural network model is set to 200 epochs.

Finally, in the volume convolutional neural network model of this embodiment, the performance of the 3D-ResNet-50 deep learning model and the 3D-DenseNet-121 deep learning model is shown in Table 1 below: model/performance Accuracy precision Recall rate (recall) AUC F1-score 3D-ResNet-50 0.82 (0.1) 0.78 (0.03) 0.72 (0.11) 0.82 (0.1) 0.75 (0.07) 3D-DenseNet-121 0.74 (0.09) 0.71 (0.07) 0.8 (0.2) 0.78 (0.14) 0.75 (0.11) Table 1. Volume Convolutional Neural Network Model Performance

Accordingly, a volumetric convolutional neural network model that can be used in the operating method of the chemotherapy brain image visualization classification system of this embodiment is trained.

Therefore, as shown in FIG. 2, first, in step (A) of this embodiment, after the processing module 100 in the chemotherapy brain image visualization classification system 10 receives at least one new original whole-brain MRI image, it will further execute In step (B), the processing module 100 will use the aforementioned image pre-processing means (i.e., Statistical Parametric Mapping (SPM)) to process as a resting-state functional magnetic resonance imaging (resting-state fMRI) At least one original whole brain MRI image. Finally, at least one whole-brain MRI image after preprocessing was obtained.

Next, in step (C), the processing module 100 also extracts at least one whole brain average low-frequency fluctuation amplitude fraction (Mean fractional amplitude of low-frequency fluctuations, mfALFF) from the at least one pre-processed whole brain MRI image. )Information. Accordingly, the pre-work before inputting the previously trained volume convolutional neural network model is completed.

Further, the volumetric convolutional neural network model of this embodiment will run through the processing module 100 . Therefore, in step (D), the processing module 100 inputs the at least one whole-brain average low-frequency fluctuation amplitude fraction information into the previously trained Volume Convolutional Neural Network (Volumetric Convolutional Neural Network) model for prediction, and obtains At least one whole brain predicts the outcome.

Finally, in step (E), the processing module 100 converts the at least one whole-brain prediction result into at least one whole-brain region image through an integrated gradients algorithm. And the image of at least one brain region of the whole brain can be directly output to the monitor to directly present the voxel-based predictions of different brain regions (including at least the frontal lobe, temporal lobe, parietal lobe, and occipital lobe) in a visual way. basal metabolic activity.

其中， i代表積分梯度的維度（dimension），x和x’依序代表原始輸入影像（Original input image）以及基線影像（Baseline image），F為該體積卷積神經網路模型；並且，該原始輸入影像為前述的至少一全腦平均低頻波動振幅度分率（Mean fractional amplitude of low-frequency fluctuations, mfALFF）資訊。 In this embodiment, the function of the integrated gradients algorithm is as follows:

Among them, i represents the dimension of the integral gradient, x and x' represent the original input image (Original input image) and the baseline image (Baseline image) in sequence, F is the volume convolutional neural network model; and, the original The input image is the aforementioned at least one whole brain mean low-frequency fluctuations (Mean fractional amplitude of low-frequency fluctuations, mfALFF) information.

In this embodiment, the baseline image (Baseline image) is implemented with all-zero 3D images (all-zero 3D images) with a resolution of 64×64×64×1. Likewise, the processing module 100 can run the integrated gradients algorithm through the TensorFlow software. Therefore, regarding the presentation of the visualized image, the resolution of the visualized image in this embodiment is also presented at 64×64×64×1. Also, the rendering threshold of gradient images is set to 0.45.

Accordingly, please refer to FIG. 3 . FIG. 3 is an image of the whole brain region predicted by the operating method of the chemotherapy brain image visualization classification system of the present invention. As shown in Figure 3, the different color scales on the right side of Figure 3 can correspond to the volumetric convolutional neural network (Volumetric Convolutional Neural Network) model and integrated gradients (integrated gradients) algorithm in this embodiment for sequential prediction and visualization The voxel-based basic metabolic activity map of each brain region in the whole brain (that is, the image of the whole brain brain region) is provided to doctors and nurses for auxiliary judgment.

But what is described above is only a preferred embodiment of the present invention, and should not limit the scope of implementation of the present invention, that is, the simple changes and modifications made according to the patent scope and description of the present invention still belong to the present invention within the scope covered.

10: Chemotherapy Brain Imaging Visual Classification System 100: Processing modules 200a: storage module 200b: storage module (A)~(E): Steps

FIG. 1 is a system architecture diagram of an embodiment of the visualized classification system for chemotherapy brain images of the present invention. Fig. 2 is a flow chart of an embodiment of the operation method of the chemotherapy brain image visualization classification system of the present invention. FIG. 3 is the image of the whole brain region predicted by the operation method of the chemotherapy brain image visualization classification system of the present invention.

(A)~(E): Steps

Claims (15)

An operating method of a visualization classification system for chemotherapy brain images, comprising: (A) acquiring and providing at least one original whole-brain MRI image by means of functional Magnetic Resonance Imaging (fMRI) to a processing module; (B) the processing module executes an image pre-processing means on the at least one original whole-brain MRI image through Statistical Parametric Mapping (SPM) to obtain at least one pre-processed whole-brain MRI image; ( C) The processing module extracts at least one whole brain average low-frequency fluctuation amplitude fraction (Mean fractional amplitude of low-frequency) from the at least one pre-processed whole brain MRI image through a band pass filter. fluctuations, mfALFF) information; (D) the processing module inputs the at least one whole-brain average low-frequency fluctuation amplitude fraction information into a volume convolutional neural network (Volumetric Convolutional Neural Network) model for prediction, and obtains at least one full-scale Brain prediction results; and (E) the processing module converts the at least one whole-brain prediction result into at least one whole-brain brain region image through an integrated gradients algorithm; wherein, the volumetric convolutional neural network model Contains 3D-ResNet-50 deep learning model and 3D-DenseNet-121 deep learning model. The operation method of the visualization classification system for chemotherapy brain images according to Claim 1, wherein the at least one original whole-brain magnetic resonance imaging image is a resting-state functional magnetic resonance imaging (resting-state fMRI). The operation method of the chemotherapy brain image visualization classification system as described in claim 1, wherein the image preprocessing means is Statistical Parametric Mapping (SPM). The operation method of the chemotherapy brain image visualization classification system as described in claim 3, wherein the statistical parameter map (Statistical Parametric Mapping, SPM) includes a slice timing correction (Slice Timing) correction step, a displacement correction (Realignment) step, a normalization (Normalization) ) step, a spatial smoothing (Smoothing) step or a combination thereof. The operation method of the visualization classification system for chemotherapy brain images as described in Claim 1, wherein the result of the image preprocessing means is to exclude the unqualified at least one original whole-brain magnetic resonance imaging image through a screening standard, and the screening standard Excluding the at least one original whole-brain MRI image in which the translation distance of the head motion exceeds 1 millimeter (mm) or the rotation angle of the head motion exceeds 1 degree. The operating method of the visualization classification system for chemotherapy brain images as described in Claim 4, wherein the normalization step is to convert the at least one original whole-brain MRI image after the displacement correction (Realignment) step into MNI space ( Standard Montreal Neurological Institute space) was used as the back of the sample brain, and was standardized with 3mm isotropic voxels (isotropic 3mm voxels). The operation method of the chemotherapy brain image visualization classification system as described in claim 4, wherein the space smoothing (Smoothing) step is a Gaussian filter with a full width at half maximum (FWHM) of 6 millimeters (mm) (Gaussian filter) to amplify the signal-to-noise ratio of the at least one original whole-brain MRI image. The operation method of the chemotherapy brain image visualization classification system as described in claim 1, wherein the at least one whole brain average low-frequency fluctuation amplitude fraction (Mean fractional amplitude of low-frequency fluctuations, mfALFF) information is passed through a band-pass filter ( band pass filter) to obtain the at least one pre-processed whole-brain MRI image with a filtering frequency of 0.01-0.12 hertz (Hz). The operation method of the chemotherapy brain image visualization classification system as described in claim 1, wherein the 3D-ResNet-50 deep learning model is to embed the compression-excitation block (SE-block) into ResNet-50 through the residual connection (Skip Connection) formed by deep learning models. The operating method of the chemotherapy brain image visualization classification system as described in claim 9, wherein the 3D-DenseNet-121 deep learning model is a dense block (dense block) that embeds the compression-excitation block (SE-block) of the DenseNet-121 deep learning model block), and the 3D-ResNet-50 deep learning model embeds the compression-excitation block (SE-block) embedded in the dense block (dense block) of the DenseNet-121 deep learning model through the residual connection (Skip Connection). The operating method of the visualization classification system for chemotherapy brain images as described in Claim 1, wherein the resolution of the at least one whole-brain region image is 64×64×64×1 voxel. The operation method of the chemotherapy brain image visualization classification system as described in Claim 1, wherein the function of the integrated gradients algorithm is as follows:

Among them, i represents the dimension of the integral gradient, x and x' represent the original input image (Original input image) and the baseline image (Baseline image) in sequence, and F is the volume convolutional neural network model; among them, the original The input image is the at least one mean fractional amplitude of low-frequency fluctuations (mfALFF) information of the whole brain. A chemotherapy brain image visualization classification system, which runs the operation method of the chemotherapy brain image visualization classification system described in claim 1, the chemotherapy brain image visualization classification system includes: the processing module; At least one storage module is connected with the processing module. Chemotherapy brain image visualization classification system as described in claim 13, wherein the processing module is a central processing unit (Central Processing Unit, CPU), a microprocessor (Microprocessor), a graphics processing unit (Graphics Processing Unit, GPU), a specific Application Specific Integrated Circuit (ASIC) or a combination thereof. Chemotherapy brain image visualization classification system as described in claim 13, wherein the at least one storage module is random access memory (RAM), read only memory (ROM), flash memory (flash memory), hard disk (hard disk), floppy disk (soft disk), database (database), or a combination thereof.

Images