CN107169535B - Deep learning classification method and device for biological multispectral image - Google Patents
Deep learning classification method and device for biological multispectral image Download PDFInfo
- Publication number
- CN107169535B CN107169535B CN201710547113.7A CN201710547113A CN107169535B CN 107169535 B CN107169535 B CN 107169535B CN 201710547113 A CN201710547113 A CN 201710547113A CN 107169535 B CN107169535 B CN 107169535B
- Authority
- CN
- China
- Prior art keywords
- deep learning
- biological
- wavelength
- multispectral image
- filter template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013135 deep learning Methods 0.000 title claims abstract description 162
- 238000000034 method Methods 0.000 title claims abstract description 95
- 238000007781 pre-processing Methods 0.000 claims abstract description 41
- 238000013528 artificial neural network Methods 0.000 claims description 38
- 238000013527 convolutional neural network Methods 0.000 claims description 38
- 230000008569 process Effects 0.000 claims description 24
- 239000011159 matrix material Substances 0.000 claims description 23
- 238000001228 spectrum Methods 0.000 claims description 21
- 238000000513 principal component analysis Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 17
- 238000010606 normalization Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 238000000638 solvent extraction Methods 0.000 claims description 9
- 238000009826 distribution Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 8
- 210000002569 neuron Anatomy 0.000 claims description 6
- 238000012163 sequencing technique Methods 0.000 claims description 5
- 230000005284 excitation Effects 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 239000004973 liquid crystal related substance Substances 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 238000012706 support-vector machine Methods 0.000 claims description 4
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000012549 training Methods 0.000 description 29
- 230000000875 corresponding effect Effects 0.000 description 25
- 230000003595 spectral effect Effects 0.000 description 15
- 230000001575 pathological effect Effects 0.000 description 7
- 206010028980 Neoplasm Diseases 0.000 description 6
- 201000011510 cancer Diseases 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000007170 pathology Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 230000002596 correlated effect Effects 0.000 description 4
- 201000010099 disease Diseases 0.000 description 4
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 210000004027 cell Anatomy 0.000 description 3
- 239000006185 dispersion Substances 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003902 lesion Effects 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 239000000090 biomarker Substances 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000002787 reinforcement Effects 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 208000002699 Digestive System Neoplasms Diseases 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000002189 fluorescence spectrum Methods 0.000 description 1
- 230000002496 gastric effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 230000008521 reorganization Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention discloses a deep learning classification method and device for biological multispectral images. The method comprises the following steps: carrying out data preprocessing on the biological multispectral image to obtain a preprocessing result; determining at least one relevant feature filter template based on the biological phenomenon relationship of interest; taking the preprocessing result as input of a deep learning frame, performing deep learning on the biological multispectral image, and adjusting the deep learning frame by using the at least one relevant characteristic filter template, so as to obtain a trained deep learning frame; and inputting the multispectral images to be classified into the trained deep learning framework to obtain the classifications corresponding to the multispectral images to be classified. The invention can realize deep learning and classification of multispectral images.
Description
Technical Field
The invention relates to a deep learning classification method and device for biological multispectral images.
Background
In the practice of research, the correct classification of tissue sections is very important. The traditional tissue section is gray level or monochromatic fluorescence, is classified mainly through cell morphology in a physical layer, is inaccurate in classification method, is not large in normal and pathological cell morphology distinction in early stage of disease, lacks tissue multiple biological phenomenon associated factor information, and cannot provide high specificity for post-treatment and classification. Color tissue slice images refer to tissue slice images with various spectral information (e.g., excitation fluorescence) or chromatographic information (e.g., stains), which are a new generation of precision medical technology in the tissue slice field, and multispectral (including hyperspectral and hyperspectral) technology is a new technology in which each pixel of an image provides light intensity values of two or more spectral wavelengths. The Fluorescence (Fluorescence) exciting multispectral biomarker or dye-stained pathological tissue section method is a newly developed method that can simultaneously measure the correlation and intensity of biological phenomena of multiple biomarkers in the same pixel region. Multispectral images are a set of images characterizing the intensities of different spectral wavelengths with their own biological complexity, and the information disclosed is the depth interrelationship of multiple biological events characterized over different spectra. In scientific practice, a set of multispectral images needs to be classified.
For a single image, classification can be achieved by adopting an image deep learning method. The intelligent classification processing of the multispectral images is still in a starting stage, the deep learning time required by the multispectral images is long, the storage space is large, the deep learning framework can be well converged by inputting a large amount of training images, the quantity of the multispectral tissue slice images at present can not reach the large data scale, and satisfactory results can not be obtained obviously according to the conventional deep learning scheme. For example, chinese patent application 201511022779.8 discloses a hyperspectral data classification method based on a multi-layer convolution network and data reorganization folding, which comprises preprocessing three-dimensional hyperspectral data to obtain a data matrix and a label vector containing effective spectral information; carrying out characteristic dimension expansion on the data matrix, and carrying out column folding recombination on the characteristic dimension to obtain a recombined three-dimensional hyperspectral data input matrix; setting structural parameters and initial values of a multi-layer convolution network; and calculating characteristics and errors layer by utilizing a forward propagation and BP algorithm, updating the network weight and bias, and continuously iterating to obtain network stability parameters, so as to finally obtain a network model and parameters which can be used for classification. As another example, chinese patent application 201610427555.3 discloses a gastrointestinal tumor microscopic hyperspectral image processing method based on convolutional neural network, which performs dimension reduction and denoising on the spectral dimension of a gastrointestinal tissue hyperspectral training image, and only retains the main component of hyperspectral data; constructing a convolutional neural network structure; and inputting training samples in each batch into a constructed convolutional neural network by adopting a batch processing method, wherein each training sample data is a hyperspectral data main component, namely a plurality of two-dimensional gray level images, the plurality of two-dimensional gray level images are equivalent to characteristic diagrams of a plurality of input layers, adopting a cross entropy function as a loss function, and training parameters in the convolutional neural network and logistic regression layer parameters according to an average loss function in the training batch by utilizing an error back propagation algorithm until the network converges. Although these schemes can develop deep learning on hyperspectral images, correlation between spectra (revealing the interconnection of biological phenomena) is not effectively utilized to improve the deep learning specificity and robustness, and fluorescence spectrum information and noise reduction processing on autofluorescence are not available, and even if a trained neural network is obtained under the condition that the number of training images is limited, the robustness and accuracy are not satisfactory.
Disclosure of Invention
In view of this, the present invention provides a method and apparatus for deep learning and classifying a biological multispectral image, which utilizes a specific pathological state to simultaneously occur along with a plurality of complex biological phenomena, and the correlation among the biological phenomena has a regional characteristic, and combines the prior knowledge of pathology to realize the deep learning and classifying of the biological multispectral image.
The invention provides a deep learning classification method of biological multispectral images, which comprises the following steps: carrying out data preprocessing on the biological multispectral image to obtain a preprocessing result; determining at least one relevant feature filter template based on the biological phenomenon relationship of interest; taking the preprocessing result as input of a deep learning frame, performing deep learning on the biological multispectral image, and adjusting the deep learning frame by using the at least one relevant characteristic filter template, so as to obtain a trained deep learning frame; and inputting the multispectral images to be classified into the trained deep learning framework to obtain the classifications corresponding to the multispectral images to be classified.
According to the deep learning classification method of the present invention, preferably, the data preprocessing includes the steps of:
(1) Converting each multispectral image into an MxN matrix, wherein M is the number of the multispectral image pixels, and N is the number of wavelength values in the multispectral image;
(2) calculating the first m wavelengths which have larger contributions to the matrix by using Principal Component Analysis (PCA);
(3) at the position ofSelecting the most relevant wavelength combination A from the wavelength combinations, wherein A is the wavelength combination containing n different wavelength values;
(4) partitioning the multispectral image, calculating the correlation coefficient of each wavelength in the wavelength combination A aiming at the image of each region, and selecting a region W corresponding to the maximum correlation coefficient;
(5) and carrying out pixel distribution histogram normalization on the intensities of different wavelengths in the wavelength combination A in the region W, and taking the normalization result as the preprocessing result.
According to the deep learning classification method of the present invention, preferably, the step (3) includes: calculating the correlation coefficient of any two wavelengths in m wavelengths, sequencing the correlation coefficients from large to small, regarding the wavelength combination containing n different wavelength values as a large group in s combinations corresponding to the largest first s correlation coefficients to obtain a plurality of large groups, adding the correlation coefficients corresponding to all the wavelength combinations in each large group to obtain a plurality of sums Q, and taking the largest sum Q max The corresponding n wavelengths form a wavelength combination a.
According to the deep learning classification method of the present invention, it is preferable that the relevant feature filter templates are designed based on correlations of various biological phenomena.
According to the deep learning classification method of the present invention, preferably, the interrelationship of the plurality of biological phenomena includes: the positional relationship of the different biological phenomena, the relative sizes of the areas occupied by the different biological phenomena and/or the distance separating the different biological phenomena.
According to the deep learning classification method of the present invention, preferably, the designing the relevant feature filter template based on the interrelationship of the plurality of biological phenomena includes:
(1) Determining the area size and the number of pixels of the relevant characteristic filter template according to the area size of the biological phenomenon occurrence area of interest;
(2) Determining a pixel depth of the relevant feature filter template based on the number of different wavelength values in the multispectral image; and/or
(3) The widths and shapes of the different components in the correlation feature filter template are determined based on the magnitudes of correlation coefficients for the different wavelengths in the multispectral image and the unique characteristics of the biological phenomenon.
According to the deep learning classification method of the present invention, preferably, the deep learning framework uses a convolutional neural network, and the adjusting the deep learning framework with the at least one relevant feature filter template includes:
(1) Taking the at least one relevant characteristic filter template and the conventional characteristic learning template as a layer 1 convolution layer of the convolution neural network;
(2) Adding the at least one relevant characteristic filter template into a d-th layer convolution layer of the convolution neural network, wherein d is a natural number which is more than 1 and less than or equal to the number of hidden layers; or alternatively
(3) The at least one relevant feature filter template is added as a new convolutional layer to the hidden layer of the convolutional neural network.
According to the deep learning classification method of the present invention, preferably, adjusting the deep learning framework with the at least one relevant feature filter template further includes: and in the deep learning process, adjusting the weight of the at least one relevant characteristic filter template.
According to the deep learning classification method of the present invention, it is preferable to use a Huber cost function as an output layer cost function of the deep learning framework.
According to the deep learning classification method of the present invention, preferably, the method further includes: in the deep learning process of the biological multispectral image, a convolution layer is randomly selected, and one or more wavelengths in all or part of the image area are not convolved by the relevant characteristic filter template.
According to the deep learning classification method of the present invention, preferably, the method further includes: in the deep learning process of the biological multispectral image, a convolution layer is randomly selected, the interested wavelength in part of the image area is not subjected to conventional template convolution, and the wavelength needing filtering is not subjected to relevant characteristic filter template convolution.
According to the deep learning classification method of the present invention, preferably, the method further includes: in the deep learning process of the biological multispectral image, one or more neuron nodes are randomly selected to be deactivated, or one or more links are randomly selected to be deactivated.
According to the deep learning classification method of the present invention, preferably, the method further includes: based on the distribution pattern of the autofluorescence, filtering out the eigenvectors of the matrix with high correlation with the autofluorescence or reducing the learning of the eigenvectors of the matrix with high correlation with the autofluorescence in the deep learning frame.
According to the deep learning classification method of the present invention, preferably, the method further includes: and in the deep learning process of the biological multispectral image, increasing the threshold value of the excitation function corresponding to the template weight of the relevant characteristic filter related to autofluorescence.
According to the deep learning classification method of the present invention, preferably, the method further includes: in the deep learning process of the biological multispectral image, when the downsampling process is carried out, the response graph of the autofluorescence wavelength characteristic template is replaced by the brightness minimum value, and the interested wavelength is replaced by the brightness maximum value, so that the learning of the spectrum wavelength related to the biological phenomenon is enhanced.
According to the deep learning classification method of the present invention, preferably, the calculating the first m wavelengths that greatly contribute to the matrix by using principal component analysis PCA includes: and calculating the contribution of each wavelength value to each pixel by using a multivariate curve resolution method MCR and a non-negative least square iterative optimization method NNLS so as to eliminate the contribution of autofluorescence in the pixel.
According to the deep learning classification method of the present invention, preferably, the deep learning framework adopts a convolutional neural network, a cyclic neural network, a deep confidence network, a deep boltzmann machine, a stacked denoising self-encoder or a support vector machine.
The invention also provides a deep learning classification device of the biological multispectral image, which comprises:
a data receiving device, a processor, a memory, and a computer program stored on the memory and executable on the processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
The data receiving device is used for receiving an input image and instructions, wherein the image comprises a biological multispectral image;
the computer program when run on the processor performs the steps of:
performing data preprocessing on the biological multispectral image received by the data receiving equipment to obtain a preprocessing result;
determining at least one relevant feature filter template according to the instruction received by the data receiving device;
taking the preprocessing result as input of a deep learning frame, performing deep learning on the biological multispectral image, and adjusting the deep learning frame by using the at least one relevant characteristic filter template so as to obtain a trained deep learning frame;
and processing the multispectral image to be classified by the trained deep learning frame to obtain the classification corresponding to the multispectral image to be classified.
According to the deep learning classification device of the present invention, preferably, the data preprocessing includes the steps of:
(1) converting each multispectral image into an MxN matrix, wherein M is the number of the multispectral image pixels, and N is the number of wavelength values in the multispectral image;
(2) calculating the first m wavelengths which have larger contributions to the matrix by using Principal Component Analysis (PCA);
(3) At the position ofSelecting the most relevant wavelength combination A from the wavelength combinations, wherein A is the wavelength combination containing n different wavelength values;
(4) partitioning the multispectral image, calculating the correlation coefficient of each wavelength in the wavelength combination A aiming at the image of each region, and selecting a region W corresponding to the maximum correlation coefficient;
(5) and carrying out pixel distribution histogram normalization on the intensities of different wavelengths in the wavelength combination A in the region W, and taking the normalization result as the preprocessing result.
According to the deep learning classifying device of the present invention, preferably, the step (3) includes: calculating the correlation coefficient of any two wavelengths in m wavelengths, sequencing the correlation coefficients from large to small, regarding the wavelength combination containing n different wavelength values as a large group in s combinations corresponding to the largest first s correlation coefficients to obtain a plurality of large groups, adding the correlation coefficients corresponding to all the wavelength combinations in each large group to obtain a plurality of sums Q, and taking the largest sum Q max The corresponding n wavelengths form a wavelength combination a.
According to the deep learning classification device of the present invention, preferably, the deep learning framework is a convolutional neural network, and the adjusting the deep learning framework with the at least one relevant feature filter template includes:
(1) Taking the at least one relevant characteristic filter template and the conventional characteristic learning template as a layer 1 convolution layer of the convolution neural network;
(2) Adding the at least one relevant characteristic filter template into a d-th layer convolution layer of the convolution neural network, wherein d is a natural number which is more than 1 and less than or equal to the number of hidden layers; or alternatively
(3) The at least one relevant feature filter template is added as a new convolutional layer to the hidden layer of the convolutional neural network.
According to the deep learning classification device of the present invention, preferably, the adjusting the deep learning framework with the at least one relevant feature filter template further includes: and adjusting the weight of the at least one relevant characteristic filter template according to the instruction received by the data receiving equipment.
The invention also provides a relevant characteristic filter template for deep learning of the biological multispectral image, which is designed based on the interrelation of various biological phenomena.
The correlation characteristic filter template according to the present invention preferably, the correlation of the plurality of biological phenomena comprises: the positional relationship of the different biological phenomena, the relative sizes of the areas occupied by the different biological phenomena and/or the distance separating the different biological phenomena.
According to the correlation characteristic filter template of the present invention, preferably, designing the correlation characteristic filter template based on the interrelationship of the plurality of biological phenomena includes:
(1) Determining the area size and the number of pixels of the relevant characteristic filter template according to the area size of the biological phenomenon occurrence area of interest;
(2) Determining a pixel depth of the relevant feature filter template based on the number of different wavelength values in the multispectral image; and/or
(3) The widths and shapes of the different components in the correlation feature filter template are determined based on the magnitudes of correlation coefficients for the different wavelengths in the multispectral image and the unique characteristics of the biological phenomenon.
According to the method, the multispectral image is subjected to data preprocessing before deep learning is started, and the designed relevant characteristic filter template is additionally added into the initial value of the neural network, so that the time of the deep learning can be greatly shortened. According to the technical scheme, in the deep learning process, the structure and the weight of the convolution layer can be adjusted according to the requirement, the learning speed is further improved, the specificity and the robustness of the deep learning are enhanced, and the multispectral classification with the maximum correlation probability is obtained.
Drawings
FIG. 1 is a block flow diagram of a method of the present invention.
FIGS. 2 (a) - (c) are schematic diagrams of a relevant feature filter template of the present invention; it shows the interrelation of one disease variant characterized by wavelength 5 and four immune phenomena present in its periphery (characterized by wavelengths 1-4, respectively); (a) the template represents early cancer stage, (b) the template represents mid cancer stage, and (c) the template represents late cancer stage.
Fig. 3 (a) - (c) are schematic diagrams of another relevant feature filter template of the present invention.
FIGS. 4 (a) - (d) are characteristic filter templates relating to labeled fluorescence and autofluorescence according to the present invention.
Fig. 5 is a correlation feature filter template characterizing spectral correlation discreteness in accordance with the present invention.
FIG. 6 is another related feature filter template of the present invention that characterizes the dispersion of spectral correlations; wherein (a) and (b) are spectral correlation templates that learn spatial basis vectors (transverse X, longitudinal Y).
FIG. 7 is a flow chart of another method of the present invention.
Fig. 8 is a block diagram of a deep learning classification device according to the present invention.
Detailed Description
The present invention will be further described with reference to specific examples, but the scope of the present invention is not limited thereto.
In the present invention, unless otherwise specified, each parameter (for example, M, N, m, n, s) indicating the number indicates a natural number.
In order to realize classification of the biological multispectral image, the application provides a deep learning classification method of the biological multispectral image shown in fig. 1 based on the inherent characteristics that the biological multispectral image can represent the depth interrelationship of different biological events, and based on the innovative thought that human intervention can be implemented in the deep learning process of the deep learning framework, and the following steps are described in detail.
< S101 > data preprocessing of biological multispectral image >
And carrying out data preprocessing on the biological multispectral image to obtain a preprocessing result. In the present application, the number of biological multispectral images required for data preprocessing is not particularly limited, and may be one or more, for example. In order to obtain a better deep learning effect, the number of biological multispectral images is a plurality, for example, more than 100, preferably more than 1000, more preferably more than 5000; for example less than 10000, preferably less than 5000, more preferably less than 1000. Thus, the deep learning effect can be ensured, and the time can be saved.
In the application, the data preprocessing comprises the following processing procedures: converting a multispectral image into an m×n matrix, wherein M is the number of pixels of the multispectral image, m=image width×image height, and N is the number of wavelength values in the multispectral image; calculating the first M wavelengths which have larger contribution to the MxN matrix by using principal component analysis PCA; wherein the calculated m wavelengths can be selected A combination of wavelengths; at->Selecting the most relevant wavelength combination A from the wavelength combinations, wherein A is the wavelength combination containing n different wavelength values; partitioning the multispectral image (for example, partitioning the multispectral image into 16 areas, namely, 4×4 total areas), calculating correlation coefficients of each wavelength in the wavelength combination A for the image of each area (here, if the spatial dispersion of the spectrum is high, and the overlapping areas of different spectrums are small, a Gaussian convolution can be firstly performed by using a Gaussian pyramid method in calculating the correlation coefficients of the areas, and then downsampling is performed to increase the correlation overlapping areas of the spectrums), and selecting an area W corresponding to the maximum correlation coefficient; the intensity of different wavelengths in the wavelength combination a in the region W is subjected to pixel distribution histogram normalization.
Further, the invention can be implemented in the following mannerThe most relevant wavelength combination A is selected from the wavelength combinations: calculating the correlation coefficient of any two wavelengths in m wavelengths, sequencing the correlation coefficients from large to small, regarding the wavelength combination containing n different wavelength values as a large group in s combinations corresponding to the largest first s correlation coefficients to obtain a plurality of large groups, adding the correlation coefficients corresponding to all the wavelength combinations in each large group to obtain a plurality of sums Q, and taking the largest sum Q max The corresponding n wavelengths form a wavelength combination a.
The data preprocessing is carried out on the biological multispectral image, so that the data dimension and the data volume required to be processed in the deep learning can be greatly simplified, and the classification accuracy of the deep learning network can be remarkably improved because the selected learning region is a plurality of regions with high spectrum correlation.
Regarding the partitioning of the multispectral image, a super pixel method (Superpixel) may also be used to obtain an accurate correlated partition, and the partition may be associated with biological significance by an image segmentation process. Preferably, the present invention employs a direct partitioning approach. This is easier to implement in practice, and direct partitioning also enhances comparability between different sets of multispectral images.
< S102, determining at least one relevant feature Filter template >
At least one relevant feature filter template is determined based on the biological phenomenon relationship of interest. In the present invention, the relevant feature filter templates may be designed based on interrelationships of various biological phenomena, such as the azimuthal relationship of different biological phenomena, the relative sizes of the areas occupied by different biological phenomena, and/or the distances separating different biological phenomena. Preferably, the design may be performed in the following manner: determining the area size and the number of pixels of the relevant characteristic filter template according to the area size of the biological phenomenon occurrence area of interest; determining a pixel depth of the relevant feature filter template based on the number of different wavelength values in the multispectral image; the widths and shapes of the different components in the correlation feature filter template are determined based on the magnitudes of correlation coefficients for the different wavelengths in the multispectral image and the unique characteristics of the biological phenomenon. This is more advantageous for efficiently completing deep learning. In addition, if a sufficient amount of multispectral image data exists, the designed relevant characteristic filter templates can be optimized through a deep learning network, or the relevant characteristic filter templates can be obtained directly through learning and training of a large amount of data.
Figures 2-4 present schematic diagrams of several relevant feature filter templates of the present invention incorporating factors that reveal the interrelation of different biological phenomena. Specifically, the three templates (a) to (c) of fig. 2 exhibit correlations of, for example, one disease variant characterized by wavelength 5 and four immune phenomena (characterized by wavelengths 1 to 4, respectively) present in the periphery thereof, the disease body being closely surrounded by the four immune phenomena. Depending on the prior knowledge and classification requirements, the templates may be further refined, with increasing area occupied by wavelength 5 in the three templates of fig. 2 (a) - (c), it being understood that (a) the template represents early stages of cancer (lesion signals are significantly less than immune signals), (b) the template represents mid-stages of cancer (lesion signals are substantially the same as immune signals), and (c) the template represents late stages of cancer (lesion signals are greater than immune signals).
The templates of fig. 3 (a) - (c) are correlated feature filter templates for two spectral relationships. The templates in FIGS. 4 (a) - (d) are relevant feature filter templates for labeled fluorescence and autofluorescence. In the present invention, the correlation feature filter templates may also characterize the spatial correlation of biological phenomena, such as the visual neurons of the human visual system that are cognized for a 45 ° angle in fig. 3 (c) and fig. 4 (d), and the visual neurons of the 90 ° angle in fig. 4 (a) - (c). Preferably, the design of the relevant feature filter templates of the present invention can be optimized using a probabilistic decision tree.
In the present invention, a relevant feature filter template for characterizing the dispersion of the spectrum interrelationship can also be designed according to the independence of the spectrum and the characteristics of the autofluorescence and other spectrums, as shown in fig. 5 and 6. Fig. 5 can be seen as a discrete split and autofluorescence of fig. 2 (a) with the "=" left end being the discrete template itself and the "=" right end being the sub-templates of different wavelengths and autofluorescence, the discrete template can be seen as a superposition of multiple sub-templates. Lambda of figure 5 1 ～λ 5 Respectively correspond to wavelengths 1 to 5 in FIG. 2, differing in lambda 1 ～λ 5 Is discretely distributed, and the gaps between the two are filled with autofluorescence. FIG. 6 is a template of a correlation feature filter with a large spectral separation, where adjacent cells of each spectrum are other spectra, representing different colors. Fig. 6 (a) and 6 (b) are spectral correlation templates for learning spatial basis vectors (lateral X, longitudinal Y), the linear combination of which can reflect spectral correlation in any direction.
It should be noted that the ten relevant feature filter templates listed in fig. 2-6 of the present invention are far from all relevant feature filter templates of the present invention, nor are they possible to list one by one for reasons of space. On the premise of conforming to biological and pathological mechanisms, various relevant characteristic filter templates can be designed according to application requirements. Under the condition of a large amount of data, the relevant characteristic filter templates can also be obtained through deep learning of a neural network, but the reality is that the current biological multispectral data volume is seriously insufficient, and the available relevant characteristic filter templates cannot be obtained, so that the invention provides a concept of designing the relevant characteristic filter templates according to pathological priori knowledge and combining a conventional deep learning method and a training model to adjust and optimize the relevant characteristic filter templates, and has important practical significance. According to the invention, the width and the shape of different components of the relevant characteristic filter template are defined and designed through the intensity of the correlation of different wavelengths, the correlation of the biological phenomena of the multispectral image is reflected, and the learning speed and the learning effect can be pertinently improved according to the application requirements.
< S103, obtaining trained deep learning frame >
And taking the preprocessing result as input of a deep learning frame, performing deep learning on the biological multispectral image, and adjusting the deep learning frame by using the at least one relevant characteristic filter template, so as to obtain a trained deep learning frame. According to the invention, a Convolutional Neural Network (CNN) deep learning framework with translational invariance can be selected, preprocessing data of the biological multispectral image can be input on the basis of a deep network weight model with conventional characteristic training, and a relevant characteristic filter template is added into the deep learning framework, so that the position of a convolutional layer where the template is positioned is reasonably adjusted, and the learning of the neural network on spectrum correlation is enhanced; meanwhile, the relevant characteristic filter template showing the spectrum correlation can be finely adjusted according to the training data, so that a template more suitable for application requirements is obtained, and the problem of insufficient data of the biological multispectral spectrum can be solved.
In the deep learning process of the present invention, at least one (i.e., one or more) relevant feature filter templates (e.g., fig. 2 to 6) and a conventional feature learning template may be used as the layer 1 convolution layer of the convolutional neural network, at least one relevant feature filter template may be added to the layer d (d is a natural number greater than 1 and less than or equal to the number of hidden layers) of the convolutional neural network, and at least one relevant feature filter template may be added as a new convolution layer to the hidden layers of the convolutional neural network. In the deep learning of the biological multispectral image, the biological phenomenon correlation reinforcement learning is targeted, and the weight of each wavelength in the relevant characteristic filter template is adjusted, so that the time of the deep learning can be greatly shortened.
< S104, image Classification >
And inputting the multispectral images to be classified into the trained deep learning framework to obtain the classifications corresponding to the multispectral images to be classified. Through steps S101-S103, preprocessing data of a biological multispectral image, selecting a maximum correlation area as input of a deep learning network, designing and obtaining a correlation characteristic filter template based on a biological pathology mechanism, taking the correlation characteristic filter template as an initial training weight, enhancing and revealing learning of spectral correlation characteristics by adjusting the weight and the relative layer position of each wavelength in the template, and finally obtaining a trained CNN neural network, wherein the trained CNN neural network comprises the trained templates which are correlation classification templates obtained by deep learning optimization adjustment of the initial input correlation characteristic filter template through training data, and the classification templates are close to the current application requirements. These trained classification templates serve as feature templates for final application in a pathology classification depth network. Then, the multispectral image to be classified can be input into a trained CNN neural network, and the classification corresponding to the multispectral image can be obtained.
It should be noted that the order of step S101 and step S102 described above may be interchanged without affecting the implementation of the present invention. According to application requirements, a relevant characteristic filter template generated by a direction gradient Histogram (HOG) can be added into the input of the neural network, so that the learning effect of the relevant characteristic filter template of the multispectral image under complex illumination and shadow change under a microscope can be improved. In the invention, pathology classification labels can be set for the training input graphs, and different training input graphs correspond to different pathology classification labels.
In addition, the neural network deep learning of the present invention can be used to improve the classification accuracy and robustness of the unprocessed and learned images by the following method:
1. the output layer cost function of the convolutional neural network uses a Huber cost function in which the L1 or L2 mode, i.e., the absolute value difference or square difference of the network calculation result and the manual calibration result, is automatically selected according to the error. The balanced processing of the Huber function may reduce sensitivity to disparate or erroneous samples in the sample while increasing the learning speed for statistically significant samples.
2. Robustness of deep network learning is improved through random training: the relevant characteristic wavelength to be weakened and learned can be selected randomly from all images of one layer in the convolution layer without template convolution or certain image areas in one layer without convolution; the wavelength to be learned can be randomly not convolved in certain image areas in a certain layer; the link and the neuron node can be directly and randomly deactivated, which is equivalent to randomly ignoring a certain node and link of the neural network in an iterative training of changing the weight for a certain time.
3. Autofluorescence is a major source of noise in multispectral, can affect the robustness of deep learning networks, and can reduce its impact to the maximum extent by the following methods:
(1) Depending on the pattern characteristics of the autofluorescence, for example the autofluorescence is spatially uniformly distributed and has similar intensities in multiple spectral dimensions, matrix eigenvectors that are highly correlated with the a priori autofluorescence are filtered out in a deep learning network or learning of matrix eigenvectors that are highly correlated is reduced.
(2) Autofluorescence has a weak correlation with all other wavelengths of light, raising the threshold of the excitation function corresponding to the relevant feature filter template weight associated with autofluorescence.
(3) In the deep learning process, in the downsampling (Pooling) process, the response graph of the autofluorescence wavelength characteristic template is replaced by the brightness minimum value, the wavelength with biological significance is replaced by the brightness maximum value, and the learning of the spectrum wavelength related to the biological phenomenon is enhanced.
(4) According to the prior knowledge obtained through experiments, the contribution of each spectrum wavelength value to each pixel can be calculated in Principal Component Analysis (PCA) by using a multivariate curve resolution Method (MCR) and non-negative least squares iterative optimization (NNLS), so that the contribution of autofluorescence is directly removed from the pixel.
The learning and classifying method of the present invention for enhancing the correlation characteristics of multiple spectra in a deep learning framework is described above based on a CNN neural network, and the method can also be applied in other deep learning frameworks, such as a cyclic neural network (recurrent neural networks, RNN), a Deep Belief Network (DBN), a deep boltzmann machine (deep boltzmann machines, DBM), a stacked denoising self-encoder (stacked denoising auto encoders, SDAE), and a support vector machine (Support Vector Machine, SVM).
According to the above method, the present invention further provides a deep learning classification device 500 for biological multispectral image, referring to fig. 8, the device 500 includes: a data receiving device 501, a processor 503, a memory 502 and a computer program stored on said memory 502 and executable on said processor 503; wherein the data receiving device 501 is configured to receive an input image and an instruction, the image comprising a biological multispectral image; the computer program when run on the processor 503 performs the steps of: performing data preprocessing on the biological multispectral image received by the data receiving device 501 to obtain a preprocessing result; determining at least one relevant feature filter template according to the instructions received by the data receiving device 501; taking the preprocessing result as input of a deep learning frame, performing deep learning on the biological multispectral image, and adjusting the deep learning frame by using the at least one relevant characteristic filter template so as to obtain a trained deep learning frame; and processing the multispectral image to be classified by the trained deep learning frame to obtain the classification corresponding to the multispectral image to be classified. The specific steps performed by the computer program when run on the processor 503 are the same as those of the learning classification method section, and the relevant content described above is incorporated herein in its entirety.
It is to be noted that, for the deep learning classification device 500 of the biological multispectral image shown in fig. 8, the connection lines among the data receiving apparatus 501, the processor 503 and the memory 502 indicate that the data signal can be transmitted in the connection line direction, such as wired transmission or wireless transmission.
Example 1
Fig. 7 is a flowchart illustrating deep learning of multiple biological multispectral images according to an embodiment of the present invention, where data preprocessing is performed on each biological multispectral image, and the data preprocessing process is as follows, taking the first biological multispectral image as an example:
the first biological multispectral image (16 wavelengths, the image of each wavelength is 256 multiplied by 256=65536) is converted into a 65536 multiplied by 16 matrix, the first m wavelengths which greatly contribute to the 65536 multiplied by 16 matrix are calculated by using principal component analysis PCA, the value of n is 2 less than or equal to n less than or equal to m in the process m, and n=3 in the process. Calculation using correlation coefficient calculation formulaThe correlation coefficient of any two wavelengths has the following calculation formula:
where h is the correlation coefficient (normalized), g is the image of wavelength 1 of the two wavelengths,for the pixel average of the wavelength 1 image, f is the image of wavelength 2 of the two wavelengths, +. >For the pixel average of the wavelength 2 image, k and l are pixel positions, k ε [1 ], image width]L E [1, image height]。
The values of the first 4 largest correlation coefficients obtained are as follows:
(1) the correlation coefficient of the first wavelength and the third wavelength is 0.9;
(2) the correlation coefficient of the fifth wavelength and the seventh wavelength is 0.8;
(3) the correlation coefficient of the fifth wavelength and the ninth wavelength is 0.7;
(4) the correlation coefficient of the third wavelength and the sixth wavelength is 0.5.
Wherein, since n=3, the combination of (1) and (2) contains four wavelength values, which is not satisfactory, and the combination of (2) and (3), (1) and (4) contains three wavelength values, which is satisfactory, and 0.8+0.7=1.5 is greater than 0.9+0.5=1.4, the combination of (2) and (3), i.e., the optimal wavelength combination a corresponding to n=3, should be selected to include the fifth wavelength, the seventh wavelength, and the ninth wavelength.
Then, according to the discrete correlation characteristic filter template of fig. 5, the first biological multispectral image is divided into 16 areas 4×4, correlation coefficients of a fifth wavelength and a seventh wavelength and correlation coefficients of a fifth wavelength and a ninth wavelength are calculated for the image of each area, an area W (for example, 13 th area in 16 areas) corresponding to the maximum correlation coefficient is obtained, and then pixel distribution histogram normalization processing is performed on intensities of different wavelengths of each spectrum in the wavelength combination a in the area W, so as to obtain a first training input image, and data preprocessing of the first biological multispectral image is completed. Similarly, the data preprocessing may be performed on the multiple biological multispectral images to obtain a second training input image and a third training input image … … nth training input image.
And inputting all training input graphs and pathology classification labels corresponding to the training input graphs into a CNN neural network, taking one (or more) related feature filter templates and conventional feature templates (such as conventional image object recognition filter feature templates obtained through image-net.org image library deep learning training) which are designed according to application requirements as a 1 st layer convolution layer of the CNN neural network deep learning framework, using the conventional feature templates for the 2 nd to d layer convolution layers, adding the related feature filter templates in the 3 rd to d+1 rd layers additionally, adjusting the weights of the related feature filter templates according to classification errors, and adjusting the weights of the conventional feature templates to strengthen the learning of spectrum correlations. In other embodiments, relevant feature filter templates may be added in layers 2-d.
Then, for a deep learning network (for example, the acceptance V3 of Google) which has been trained by adjusting the weight of a conventional feature template through conventional image learning, the classification layer needs to be readjusted according to the classification requirement of multispectral application (for example, 1000 classifications of the original acceptance V3 trained by image-net. Org can be changed to 5 to distinguish 5 different pathological classifications), the error of a multispectral image training sample learning classification result and a manual classification result is calculated, whether the error is smaller than an expected error is judged, and if the error is smaller than the expected error, classification learning is completed, so that a trained CNN neural network is obtained; if the difference is not smaller than the expected difference, the weight of the relevant characteristic filter template is required to be readjusted, and meanwhile, the weight of a part of conventional characteristic templates can be finely adjusted until the error between the multispectral image learning classification result and the artificial classification structure is smaller than the expected error, and finally the trained CNN neural network is obtained. When the method is used, a biological multispectral image is input into a trained CNN neural network, and the corresponding pathological classification can be obtained.
When the weight of the relevant characteristic filter template is adjusted, a chain rule (chain rule) inverse direction derivative is used, partial differential derivative gradients of the final output errors corresponding to each layer of connecting weight are calculated, the product of a cost function and partial differential derivative values at a learning node is used as an adjustment value of the learning weight in the maximum gradient direction, and the solving of the gradients and the inverse error propagation adjustment can be automatically completed in Keras, tensorFlow, torch or other machine learning software tools. The neuron weights are adjusted by inverse error propagation for each iteration, thereby reducing the final output error. Finally, the output classification layer uses a Softmax function and a cross entropy cost function, or normalized exponential function, to obtain probabilities of a plurality of different classes and simplify derivative operation. Iterative learning may be terminated when the classification error rate of the different categories is below the training expectations.
Example 2
This embodiment differs from embodiment 1 in that: in example 1, the layer 2 to layer d convolution layers use conventional feature templates; in this embodiment, the 2 nd to d th convolution layers use conventional feature templates and related feature filter templates to highlight reinforcement learning of spectral correlations. The rest of the settings are the same as those of embodiment 1.
Example 3
This embodiment requires the use of a reference to λ in FIG. 5 1 And lambda (lambda) 2 After inputting all training input graphs into CNN neural network, adding lambda into 1 st layer, 3 rd layer and d rd layer of CNN neural network 1 And lambda (lambda) 2 Is capable of strengthening the correlation characteristic filter template of lambda 1 And lambda (lambda) 2 Is a deep learning of the interrelationship of (a) is provided. The rest of the settings are the same as those of embodiment 1.
Example 4
This embodiment requires the use of a reference to λ in FIG. 5 1 And lambda (lambda) 2 After inputting all training input images into CNN neural network, adding lambda in the middle (or last) of CNN neural network 1 And lambda (lambda) 2 Can strengthen the pair lambda 1 And lambda (lambda) 2 Is a deep learning of the interrelationship of (a) is provided. The rest of the settings are the same as those of embodiment 1.
Example 5
The present embodiment differs from embodiment 1 in that a Recurrent Neural Network (RNN) is employed as a deep learning framework. The rest of the settings are the same as those of embodiment 1.
Example 6
In this embodiment, a multispectral imager is used to generate a biological multispectral image of a tissue sample, and the biological multispectral images are uploaded to a remote server or cloud end through a wireless communication module, a computer program is stored in the server or cloud end, and when the computer program is run, the process in embodiment 1 is executed, where the relevant feature filter template is stored in the server or in the cloud end. When determining relevant feature filter templates according to application needs, the relevant feature filter templates of interest can be selected through a device connected with the cloud in a wireless mode or a device connected with a server in a wireless (or wired mode) (such as a CCD, sCMOS camera, mobile phone, tablet computer, computer keyboard and/or mouse). When the error between the multispectral image learning classification result and the artificial classification structure in the program is smaller than the expected error, the training of the CNN neural network is completed, the trained CNN neural network is stored in a server or a cloud, and the trained CNN neural network can be stored in other storage media. And inputting the multispectral images to be classified into a trained CNN neural network, and operating the CNN neural network to obtain the corresponding classifications.
Example 7
In this embodiment, the computer instruction for performing deep learning on the biological multispectral image is stored in the mobile hard disk, and the server may call the computer instruction in the mobile hard disk or upload the computer instruction in the mobile hard disk to the server. The flow of execution of the computer instructions by the server is the same as in embodiment 1.
The present invention is not limited to the above-described embodiments, and any modifications, improvements, substitutions, and the like, which may occur to those skilled in the art, fall within the scope of the present invention without departing from the spirit of the invention.
Claims (23)
1. A deep learning classification method for biological multispectral images, which is characterized by comprising the following steps: carrying out data preprocessing on the biological multispectral image to obtain a preprocessing result; determining at least one relevant feature filter template based on the biological phenomenon relationship of interest; taking the preprocessing result as input of a deep learning frame, performing deep learning on the biological multispectral image, and adjusting the deep learning frame by using the at least one relevant characteristic filter template, so as to obtain a trained deep learning frame; inputting the multispectral image to be classified into the trained deep learning frame to obtain the classification corresponding to the multispectral image to be classified;
The data preprocessing comprises the following steps:
(1) converting each multispectral image into an MxN matrix, wherein M is the number of the multispectral image pixels, and N is the number of wavelength values in the multispectral image;
(2) calculating the first m wavelengths which have larger contributions to the matrix by using Principal Component Analysis (PCA);
(3) at the position ofSelecting the most relevant wavelength combination A from the wavelength combinations, wherein A is the wavelength combination containing n different wavelength values;
(4) partitioning the multispectral image, calculating the correlation coefficient of each wavelength in the wavelength combination A aiming at the image of each region, and selecting a region W corresponding to the maximum correlation coefficient;
(5) and carrying out pixel distribution histogram normalization on the intensities of different wavelengths in the wavelength combination A in the region W, and taking the normalization result as the preprocessing result.
2. The deep learning categorization method of claim 1, wherein step (3) comprises: calculating the correlation coefficient of any two wavelengths in m wavelengths, sequencing the correlation coefficients from large to small, regarding the wavelength combination containing n different wavelength values as a large group in s combinations corresponding to the largest first s correlation coefficients to obtain a plurality of large groups, adding the correlation coefficients corresponding to all the wavelength combinations in each large group to obtain a plurality of sums Q, and taking the largest sum Q max The corresponding n wavelengths form a wavelength combination a.
3. The deep learning classification method of claim 1, wherein the correlation feature filter templates are designed based on correlations of multiple biological phenomena.
4. The deep learning classification method of claim 3, wherein the interrelationship of the plurality of biological phenomena comprises: the positional relationship of the different biological phenomena, the relative sizes of the areas occupied by the different biological phenomena and/or the distance separating the different biological phenomena.
5. The deep learning classification method of claim 3, wherein the designing the relevant feature filter templates based on interrelationships of a plurality of biological phenomena comprises:
(1) Determining the area size and the number of pixels of the relevant characteristic filter template according to the area size of the biological phenomenon occurrence area of interest;
(2) Determining a pixel depth of the relevant feature filter template based on the number of different wavelength values in the multispectral image; and/or
(3) The widths and shapes of the different components in the correlation feature filter template are determined based on the magnitudes of correlation coefficients for the different wavelengths in the multispectral image and the unique characteristics of the biological phenomenon.
6. The deep learning classification method of claim 1, wherein the deep learning framework employs a convolutional neural network, and wherein adjusting the deep learning framework with the at least one relevant feature filter template comprises:
(1) Taking the at least one relevant characteristic filter template and the conventional characteristic learning template as a layer 1 convolution layer of the convolution neural network;
(2) Adding the at least one relevant characteristic filter template into a d-th layer convolution layer of the convolution neural network, wherein d is a natural number which is more than 1 and less than or equal to the number of hidden layers; or alternatively
(3) The at least one relevant feature filter template is added as a new convolutional layer to the hidden layer of the convolutional neural network.
7. The deep learning classification method of claim 1, wherein adjusting the deep learning framework with the at least one relevant feature filter template further comprises: and in the deep learning process, adjusting the weight of the at least one relevant characteristic filter template.
8. The deep learning classification method of claim 1, wherein a Huber cost function is used as the output layer cost function of the deep learning framework.
9. The deep learning classification method of claim 1, further comprising: in the deep learning process of the biological multispectral image, a convolution layer is randomly selected, and one or more wavelengths in all or part of the image area are not convolved by the relevant characteristic filter template.
10. The deep learning classification method of claim 1, further comprising: in the deep learning process of the biological multispectral image, a convolution layer is randomly selected, the interested wavelength in part of the image area is not subjected to conventional template convolution, and the wavelength needing filtering is not subjected to relevant characteristic filter template convolution.
11. The deep learning classification method of claim 1, further comprising: in the deep learning process of the biological multispectral image, one or more neuron nodes are randomly selected to be deactivated, or one or more links are randomly selected to be deactivated.
12. The deep learning classification method of claim 1, further comprising: based on the distribution pattern of the autofluorescence, filtering out the eigenvectors of the matrix with high correlation with the autofluorescence or reducing the learning of the eigenvectors of the matrix with high correlation with the autofluorescence in the deep learning frame.
13. The deep learning classification method of claim 1, further comprising: and in the deep learning process of the biological multispectral image, increasing the threshold value of the excitation function corresponding to the template weight of the relevant characteristic filter related to autofluorescence.
14. The deep learning classification method of claim 1, further comprising: in the deep learning process of the biological multispectral image, when the downsampling process is carried out, the response graph of the autofluorescence wavelength characteristic template is replaced by the brightness minimum value, and the interested wavelength is replaced by the brightness maximum value, so that the learning of the spectrum wavelength related to the biological phenomenon is enhanced.
15. The deep learning classification method of claim 1, wherein the calculating the first m wavelengths that contribute more to the matrix using principal component analysis PCA comprises: and calculating the contribution of each wavelength value to each pixel by using a multivariate curve resolution method MCR and a non-negative least square iterative optimization method NNLS so as to eliminate the contribution of autofluorescence in the pixel.
16. The deep learning classification method of claim 1, wherein the deep learning framework employs a convolutional neural network, a cyclic neural network, a deep belief network, a deep boltzmann machine, a stacked de-noising self-encoder, or a support vector machine.
17. A deep learning classification device for biological multispectral images, the device comprising:
a data receiving device, a processor, a memory, and a computer program stored on the memory and executable on the processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the data receiving device is used for receiving an input image and instructions, wherein the image comprises a biological multispectral image;
the computer program when run on the processor performs the steps of:
performing data preprocessing on the biological multispectral image received by the data receiving equipment to obtain a preprocessing result;
determining at least one relevant feature filter template according to the instruction received by the data receiving device;
taking the preprocessing result as input of a deep learning frame, performing deep learning on the biological multispectral image, and adjusting the deep learning frame by using the at least one relevant characteristic filter template so as to obtain a trained deep learning frame;
processing the multispectral image to be classified by the trained deep learning frame to obtain the classification corresponding to the multispectral image to be classified;
the data preprocessing comprises the following steps:
(1) Converting each multispectral image into an MxN matrix, wherein M is the number of the multispectral image pixels, and N is the number of wavelength values in the multispectral image;
(2) calculating the first m wavelengths which have larger contributions to the matrix by using Principal Component Analysis (PCA);
(3) at the position ofSelecting the most relevant wavelength combination A from the wavelength combinations, wherein A is the wavelength combination containing n different wavelength values;
(4) partitioning the multispectral image, calculating the correlation coefficient of each wavelength in the wavelength combination A aiming at the image of each region, and selecting a region W corresponding to the maximum correlation coefficient;
(5) and carrying out pixel distribution histogram normalization on the intensities of different wavelengths in the wavelength combination A in the region W, and taking the normalization result as the preprocessing result.
18. The deep learning classification device of claim 17, wherein step (3) comprises: calculating the correlation coefficient of any two wavelengths in m wavelengths, sequencing the correlation coefficients from large to small, regarding the wavelength combination containing n different wavelength values as a large group in s combinations corresponding to the largest first s correlation coefficients to obtain a plurality of large groups, adding the correlation coefficients corresponding to all the wavelength combinations in each large group to obtain a plurality of sums Q, Take the maximum sum Q max The corresponding n wavelengths form a wavelength combination a.
19. The deep learning classification device of claim 17, wherein the deep learning framework is a convolutional neural network, the adjusting the deep learning framework with the at least one relevant feature filter template comprising:
(1) Taking the at least one relevant characteristic filter template and the conventional characteristic learning template as a layer 1 convolution layer of the convolution neural network;
(2) Adding the at least one relevant characteristic filter template into a d-th layer convolution layer of the convolution neural network, wherein d is a natural number which is more than 1 and less than or equal to the number of hidden layers; or alternatively
(3) The at least one relevant feature filter template is added as a new convolutional layer to the hidden layer of the convolutional neural network.
20. The deep learning classification device of claim 19, wherein the adjusting the deep learning framework with the at least one relevant feature filter template further comprises: and adjusting the weight of the at least one relevant characteristic filter template according to the instruction received by the data receiving equipment.
21. A correlation feature filter template for use in the deep learning classification method of biological multispectral images of any one of claims 1 to 16, wherein the correlation feature filter template is designed based on interrelationships of multiple biological phenomena.
22. The correlation filter template of claim 21, wherein the interrelationship of the plurality of biological phenomena comprises: the positional relationship of the different biological phenomena, the relative sizes of the areas occupied by the different biological phenomena and/or the distance separating the different biological phenomena.
23. The correlation feature filter template of claim 21, wherein designing the correlation feature filter template based on correlations of multiple biological phenomena comprises:
(1) Determining the area size and the number of pixels of the relevant characteristic filter template according to the area size of the biological phenomenon occurrence area of interest;
(2) Determining a pixel depth of the relevant feature filter template based on the number of different wavelength values in the multispectral image; and/or
(3) The widths and shapes of the different components in the correlation feature filter template are determined based on the magnitudes of correlation coefficients for the different wavelengths in the multispectral image and the unique characteristics of the biological phenomenon.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710547113.7A CN107169535B (en) | 2017-07-06 | 2017-07-06 | Deep learning classification method and device for biological multispectral image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710547113.7A CN107169535B (en) | 2017-07-06 | 2017-07-06 | Deep learning classification method and device for biological multispectral image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107169535A CN107169535A (en) | 2017-09-15 |
| CN107169535B true CN107169535B (en) | 2023-11-03 |
Family
ID=59822924
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710547113.7A Active CN107169535B (en) | 2017-07-06 | 2017-07-06 | Deep learning classification method and device for biological multispectral image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107169535B (en) |
Families Citing this family (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9947103B1 (en) * | 2017-10-03 | 2018-04-17 | StradVision, Inc. | Learning method and learning device for improving image segmentation and testing method and testing device using the same |
| US10095977B1 (en) * | 2017-10-04 | 2018-10-09 | StradVision, Inc. | Learning method and learning device for improving image segmentation and testing method and testing device using the same |
| CN107657250B (en) * | 2017-10-30 | 2020-11-24 | 四川理工学院 | Bearing fault detection and positioning method and detection and positioning model implementation system and method |
| CN107832797B (en) * | 2017-11-17 | 2020-04-07 | 西安电子科技大学 | Multispectral image classification method based on depth fusion residual error network |
| CN108230335B (en) * | 2017-12-29 | 2020-12-01 | 北京数字精准医疗科技有限公司 | Multispectral fluorescence segmentation method and device |
| CN108460342B (en) * | 2018-02-05 | 2021-01-01 | 西安电子科技大学 | Hyperspectral image classification method based on convolutional neural network and cyclic neural network |
| JP7137935B2 (en) * | 2018-02-27 | 2022-09-15 | シスメックス株式会社 | Image analysis method, image analysis device, program, method for manufacturing trained deep learning algorithm, and trained deep learning algorithm |
| CN108734199B (en) * | 2018-04-24 | 2021-09-07 | 西北工业大学 | Hyperspectral image robust classification method based on segmented depth features and low-rank representation |
| CN108760668B (en) * | 2018-06-01 | 2019-10-29 | 南京林业大学 | Pinus massoniana Seedlings root moisture method for fast measuring based on weighting autocoder |
| CN109191564B (en) * | 2018-07-27 | 2020-09-04 | 中国科学院自动化研究所 | Depth learning-based three-dimensional reconstruction method for fluorescence tomography |
| CN109724917B (en) * | 2018-12-26 | 2021-01-22 | 西安电子科技大学 | Light splitting method based on neural network and broadband filtering modulation |
| CN113614498A (en) * | 2019-02-06 | 2021-11-05 | 日本电气株式会社 | Filter learning apparatus, filter learning method, and non-transitory computer readable medium |
| CN109918513B (en) * | 2019-03-12 | 2023-04-28 | 北京百度网讯科技有限公司 | Image processing method, device, server and storage medium |
| CN110646350B (en) * | 2019-08-28 | 2023-06-02 | 深圳数联天下智能科技有限公司 | Product classification method, device, computing equipment and computer storage medium |
| CN110738659A (en) * | 2019-12-23 | 2020-01-31 | 中电健康云科技有限公司 | analysis method of micro-fluorescence hyperspectral image based on deconvolution and unmixing |
| CN112086199B (en) * | 2020-09-14 | 2023-06-09 | 中科院计算所西部高等技术研究院 | Liver cancer data processing system based on multiple groups of study data |
| CN115245312A (en) * | 2021-04-27 | 2022-10-28 | 山东威高宏瑞医学科技有限公司 | Endoscope multispectral image processing system and processing and training method |
| CN113111863B (en) * | 2021-05-13 | 2023-11-24 | 辽宁工程技术大学 | Graph convolution network hyperspectral classification method based on micro batch processing and cascade fusion |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015168026A2 (en) * | 2014-04-28 | 2015-11-05 | The Broad Institute, Inc. | Method for label-free image cytometry |
| WO2017044782A1 (en) * | 2015-09-11 | 2017-03-16 | EyeVerify Inc. | Image and feature quality, image enhancement and feature extraction for ocular-vascular and facial recognition, and fusing ocular-vascular with facial and/or sub-facial information for biometric systems |
| CN106815601A (en) * | 2017-01-10 | 2017-06-09 | 西安电子科技大学 | Hyperspectral image classification method based on recurrent neural network |
| CN106845418A (en) * | 2017-01-24 | 2017-06-13 | 北京航空航天大学 | A kind of hyperspectral image classification method based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10776606B2 (en) * | 2013-09-22 | 2020-09-15 | The Regents Of The University Of California | Methods for delineating cellular regions and classifying regions of histopathology and microanatomy |
-
2017
- 2017-07-06 CN CN201710547113.7A patent/CN107169535B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015168026A2 (en) * | 2014-04-28 | 2015-11-05 | The Broad Institute, Inc. | Method for label-free image cytometry |
| WO2017044782A1 (en) * | 2015-09-11 | 2017-03-16 | EyeVerify Inc. | Image and feature quality, image enhancement and feature extraction for ocular-vascular and facial recognition, and fusing ocular-vascular with facial and/or sub-facial information for biometric systems |
| CN106815601A (en) * | 2017-01-10 | 2017-06-09 | 西安电子科技大学 | Hyperspectral image classification method based on recurrent neural network |
| CN106845418A (en) * | 2017-01-24 | 2017-06-13 | 北京航空航天大学 | A kind of hyperspectral image classification method based on deep learning |
Non-Patent Citations (2)
| Title |
|---|
| 基于多光谱大数据分析的智能森林火警检测;卢柯楠;胡晓惠;李小涛;;中国安全防范认证(03);全文 * |
| 高光谱遥感影像分类研究进展;杜培军;夏俊士;薛朝辉;谭琨;苏红军;鲍蕊;;遥感学报(02);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107169535A (en) | 2017-09-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107169535B (en) | Deep learning classification method and device for biological multispectral image | |
| CN111368896B (en) | Hyperspectral remote sensing image classification method based on dense residual three-dimensional convolutional neural network | |
| CN109754017B (en) | Hyperspectral image classification method based on separable three-dimensional residual error network and transfer learning | |
| CN111191736B (en) | Hyperspectral image classification method based on depth feature cross fusion | |
| CN110728224B (en) | Remote sensing image classification method based on attention mechanism depth Contourlet network | |
| JP6539901B2 (en) | PLANT DISEASE DIAGNOSTIC SYSTEM, PLANT DISEASE DIAGNOSTIC METHOD, AND PROGRAM | |
| WO2019091459A1 (en) | Image processing method, processing apparatus and processing device | |
| CN110033032B (en) | Tissue slice classification method based on microscopic hyperspectral imaging technology | |
| CN115272196B (en) | Method for predicting focus area in histopathological image | |
| CN108229551B (en) | Hyperspectral remote sensing image classification method based on compact dictionary sparse representation | |
| CN114119444A (en) | Multi-source remote sensing image fusion method based on deep neural network | |
| CN109801275B (en) | Potato disease detection method and system based on image recognition | |
| CN112308152A (en) | Hyperspectral image ground object classification method based on spectrum segmentation and homogeneous region detection | |
| CN113344077A (en) | Anti-noise solanaceae disease identification method based on convolution capsule network structure | |
| CN117015796A (en) | Method for processing tissue images and system for processing tissue images | |
| Jenifa et al. | Classification of cotton leaf disease using multi-support vector machine | |
| Deb et al. | LS-Net: A convolutional neural network for leaf segmentation of rosette plants | |
| CN113256733B (en) | Camera spectral sensitivity reconstruction method based on confidence voting convolutional neural network | |
| CN113096023A (en) | Neural network training method, image processing method and device, and storage medium | |
| Yin et al. | Adaptive low light visual enhancement and high-significant target detection for infrared and visible image fusion | |
| Khan et al. | An effective approach to address processing time and computational complexity employing modified CCT for lung disease classification | |
| CN117152179A (en) | Segmentation and classification method for realizing rectal cancer CT image based on U-Net and SENet | |
| Krishna et al. | Stain normalized breast histopathology image recognition using convolutional neural networks for cancer detection | |
| CN110866552A (en) | Hyperspectral image classification method based on full convolution space propagation network | |
| Zheng et al. | Stain standardization capsule: A pre-processing module for histopathological image analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |