WO2019174376A1 - 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法 - Google Patents

一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法 Download PDF

Info

Publication number
WO2019174376A1
WO2019174376A1 PCT/CN2019/070588 CN2019070588W WO2019174376A1 WO 2019174376 A1 WO2019174376 A1 WO 2019174376A1 CN 2019070588 W CN2019070588 W CN 2019070588W WO 2019174376 A1 WO2019174376 A1 WO 2019174376A1
Authority
WO
WIPO (PCT)
Prior art keywords
network
lung
image
information
channel
Prior art date
Application number
PCT/CN2019/070588
Other languages
English (en)
French (fr)
Inventor
徐睿
叶昕辰
林林
李豪杰
樊鑫
罗钟铉
Original Assignee
大连理工大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 大连理工大学 filed Critical 大连理工大学
Priority to US16/649,650 priority Critical patent/US11170502B2/en
Publication of WO2019174376A1 publication Critical patent/WO2019174376A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/40—Analysis of texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/40—Analysis of texture
    • G06T7/41—Analysis of texture based on statistical description of texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30061—Lung

Definitions

  • the invention belongs to the field of medical image processing and computer vision, and relates to extracting relevant features of apparent and geometric information of lung computed tomography (CT) images by using a deep neural network framework, and classifying different types of lung CT image textures. Specifically, it relates to a lung texture recognition method based on deep neural network to extract apparent and geometric features.
  • CT computed tomography
  • Diffuse lung disease is a general term for lung diseases that exhibit a wide distribution of large-area lung shadows in the lung region of CT images. Because of the complex texture of these lungs, it is confusing, and even for experienced radiologists, it is difficult to accurately identify different lung textures. Therefore, there is a need to establish a computer-aided diagnosis (CAD) system for accurate and efficient automatic identification of lung texture in CT images of diffuse lung disease.
  • CAD computer-aided diagnosis
  • a key technique for establishing such a CAD system is the accurate and efficient automatic identification of lung textures in any region of interest (ROI) of the lung region in a CT image.
  • Traditional CT image lung texture recognition is usually based on a two-step method, that is, the feature quantity that can characterize the texture of the lung is first designed manually, and then the classifier that can effectively distinguish the feature quantity is trained. Because the techniques of classifier training are relatively mature, most methods often use existing methods, such as neural networks, support vector machines, K-nearest neighbor classifiers, etc.; therefore, researchers will focus on design to fully characterize lung texture.
  • the characteristic quantity of the characteristic for example, a feature pocket-based lung texture recognition method (R. Xu, Y. Hirano, R. Tachibana, and S. Kido, "Classification of diffuse lung disease patterns on high-resolution computed tomography by a bag of words approach, "in International Conference on Medical Image Computing & Computer-assisted Intervention (MICCAI), 2011, p. 183.).
  • deep neural networks have revolutionized the field of image processing and computer vision. It combines the two steps of the traditional image recognition method into one, that is, the feature quantity design and the classifier training are merged into a whole end-to-end framework, and the feature quantity which can effectively represent the data is directly mined and learned from the image data, and Apply it to image recognition tasks. Based on this model, deep learning has demonstrated superior performance in terms of natural image recognition, face recognition, and the like, compared to conventional image recognition methods. Recently, deep learning has also been applied to lung texture recognition.
  • CNN convolutional neural networks
  • the deep learning method has been applied to the lung texture recognition of CT images, and compared with the traditional recognition method, there is a certain improvement in the recognition accuracy, but compared with the actual needs of the CAD system, the lung texture recognition There is still a certain gap in accuracy.
  • the above method based on deep learning has two problems. First, the number of network layers constructed is small, and it is impossible to learn the feature quantity of the lung texture efficiently, and the advantages of deep learning cannot be fully utilized. Secondly, according to the particularity of the lung texture image, different lung textures are not only reflected in the apparent difference reflected by the pixel's light and dark changes, but also reflected in the geometrical differences of the texture microstructure. The above method based on deep learning only excavates the apparent information of the lung texture, ignoring its geometric features, and thus fails to fully learn the effective features, and cannot complete high-precision texture recognition.
  • the present invention designs a lung texture recognition framework based on deep neural network to extract apparent and geometric features, which uses a residual network (Kaiming He, Xiangyu Zhang, and et. al., "Deep residual learning for The residual module in image recognition, "In Computer Vision and Pattern Recognition, 2016, pp. 770-778.”, builds an 18-layer dual-channel residual network. It has a two-channel network structure, which can extract and learn the apparent and geometric features of the lung texture, and organically fuse it to achieve high-precision automatic recognition of lung texture.
  • a residual network Kaiming He, Xiangyu Zhang, and et. al., "Deep residual learning for The residual module in image recognition, "In Computer Vision and Pattern Recognition, 2016, pp. 770-778.”
  • the present invention aims to overcome the deficiencies of the prior art, and provides a lung texture recognition method based on deep neural network to extract apparent and geometric features.
  • the method has a deep two-channel network structure, which can separately extract the apparent and geometric information of the lung texture and organically fuse it to achieve high-precision automatic recognition of the lung texture.
  • a specific technical solution of the present invention is a lung texture recognition method for extracting apparent and geometric features based on a deep neural network, comprising the following steps:
  • the initial data includes lung texture CT image patches, corresponding geometric information image patches, and corresponding category labels for training, validation, and testing.
  • step (2) Based on step (2) to obtain a two-channel residual network for training.
  • the construction of the dual channel residual network includes the following steps:
  • the constructed two-channel residual network consists of 18 layers, consisting of 15 convolutional layers and 3 fully connected layers; the upper channel uses (32 ⁇ 32 ⁇ 1 CT image patches) image appearance information as Input, the lower channel uses the corresponding (32 ⁇ 32 ⁇ 3 geometric information image small block) image geometry information as input;
  • the jump structure calculates the residuals of the convolutional layer input information and the output information after the convolutional layer is calculated.
  • the network can effectively deepen the network layer by learning the residuals, avoiding the gradient disappearance and the gradient explosion problem.
  • the network can learn efficiently; the residual calculation formula is as follows:
  • y denotes the final output of the convolutional layer
  • F denotes the convolution calculation function
  • x denotes the input of the network
  • i denotes the number of convolutional layer indexes
  • the domain of definition is [1, 15]
  • W i denotes the i-th volume
  • the coefficient of the layer is obtained by network training;
  • a maximum pooling layer is added after the first convolution layer and after the last convolution layer;
  • the convolution step size of the sixth, tenth, and fourteenth convolutional layers is set to 2 (the default is 1), which serves as a double sampling down, and the network is continuously expanded. Feeling wild.
  • step (2) The training is performed based on the two-channel residual network obtained in step (2), which specifically includes the following steps:
  • L( ⁇ ) represents the loss function value
  • n represents the number of samples participating in the training in the batch.
  • n is 30
  • x represents the data matrix participating in the training in the batch
  • ⁇ is the summation operator
  • y′ Represents the actual category label matrix corresponding to x
  • log( ⁇ ) represents a logarithm operation
  • y represents a category label matrix of x obtained by network classification.
  • the invention is based on the idea of deep learning, and constructs an 18-layer dual-channel residual network by using the jump structure in the residual network.
  • the inputs to the two channels are the original image and the corresponding geometric information image.
  • the network Through the network, a classification result with a higher correct recognition rate can be obtained.
  • the system has the following features:
  • the system is easy to construct. It only relies on the original CT image small block and the corresponding geometric information image small block, and the classification result with higher correct recognition rate can be obtained through the dual channel residual network;
  • the number of layers of the dual-channel residual network is deepened while ensuring that the network can learn normally, and the ability of the network to mine image features is improved.
  • Figure 1 is a flow chart of a specific implementation.
  • Figure 2 is an example of a CT image of a type 7 lung texture, in which (a) nodular; (b) emphysema; (c) honeycomb; (d) fixed; (e) frosted glass; (f) normal; (g) frosted glass with lines.
  • Figure 3 is a lung texture map in which (a) a small portion of the CT image; (b) an example of the extracted geometric information map.
  • Figure 4 is a simplified diagram of the network structure.
  • Figure 5 is a comparison of the correctness rate of classification results with other methods, (a) LeNet-5 model recognition accuracy; (b) Bag-of-Feature model recognition accuracy; (c) 5-layer convolution Neural network (CNN-5) model recognition accuracy; (d) single residual network (ResNet-18) model recognition accuracy; (e) the present invention (DB-ResNet-18) recognition accuracy.
  • Figure 6 is a comparison of the chaotic matrix of the classification results with other methods, (a) the chaos matrix of the Bag-of-Feature model; (b) the confusion matrix of the 5-layer convolutional neural network (CNN-5) model; a single residual network (ResNet-18) model chaos matrix; (d) the inventive (DB-ResNet-18) chaotic matrix.
  • the invention provides a lung texture recognition method based on deep neural network for extracting apparent and geometric features, which is described in detail below with reference to the accompanying drawings and embodiments:
  • the invention constructs a two-channel residual network, and uses the CT image of the lung to perform training, and achieves a high correct recognition rate in the test.
  • the specific implementation process is shown in FIG. 1 , and the method includes the following steps;
  • a total of 217 lung images of the lungs were collected in the experiment.
  • CT images of 187 patients included six typical textures of diffuse lung disease, namely nodular, emphysema, honeycomb, fixed, frosted glass and stripped ground glassy texture; CT of the remaining 30 patients In the image, only the normal lung tissue texture is presented.
  • 7 lung texture image patches were generated for the experiment.
  • the CT image is sampled on the corresponding slice of the CT image using a 32 ⁇ 32 ⁇ 1 scan frame. Starting from the upper left corner of the corresponding slice, the scanning is performed in steps of 8 pixels in the horizontal and vertical directions, respectively.
  • the center point of the search box falls within the marked area, the CT image area in the search box is saved, and the lung texture category of the corresponding CT image area is recorded, and finally a series of 7 types of typical sizes of 32 ⁇ 32 ⁇ 1 are obtained.
  • a small piece of lung texture CT image is shown in Figure 2 shows an example of the resulting seven lung texture CT image patches.
  • a size of 3 ⁇ 3 Hessian matrix is calculated according to the following formula.
  • H represents the Hessian matrix
  • I represents the gray value of the CT image, which is [0, 255].
  • the eigenvalue decomposition of the matrix is performed, and three eigenvalues are obtained at each pixel.
  • These feature values are arranged according to the position of the pixel points, and are reconstructed into an image of 32 ⁇ 32 ⁇ 3, which reflects the geometric features of the original 32 ⁇ 32 ⁇ 1 CT image patches, which are corresponding geometric information image patches.
  • Figure 3 shows an example of a CT image tile and a corresponding geometric information image tile.
  • steps (1-3) and (1-4) Through steps (1-3) and (1-4), a total of 72348 sets of data are obtained, each set of data including a 32 ⁇ 32 ⁇ 1 CT image patch and a 32 ⁇ 32 ⁇ 3 geometric information image. A small block and a category label for the corresponding lung texture.
  • the 54392 groups were randomly selected as the training set and the verification set to train and improve the two-channel residual network. The remaining 17956 groups were used as test sets to test the network with the model and parameters determined.
  • the training set, verification set, and test set are independent of each other.
  • the two-channel residual network constructed by the present invention has 18 layers, consisting of 15 convolution layers and 3 fully connected layers.
  • the upper channel uses 32 x 32 x 1 CT image patches (image apparent information) as input, and the lower channel uses the corresponding 32 x 32 x 3 geometric information image patches (image geometry information) as input.
  • the jump structure calculates the residuals of the convolutional layer input information and the output information after the convolutional layer is calculated.
  • the network can effectively deepen the network layer by learning the residuals, avoiding the gradient disappearance and the gradient explosion problem.
  • the network can learn efficiently.
  • the residual calculation formula is as follows:
  • y denotes the final output of the convolutional layer
  • F denotes the convolution calculation function
  • x denotes the input of the network
  • i denotes the number of convolutional layer indexes
  • the domain of definition is [1, 15]
  • W i denotes the i-th volume
  • the coefficient of the layer is obtained by network training.
  • a maximum pooling layer is added after the first convolution layer and after the last convolution layer.
  • the convolution step size of the sixth, tenth, and fourteenth convolutional layers is set to 2 (the default is 1), which serves as a double sampling down, and the network is continuously expanded. Feeling wild.
  • step (2) Based on step (2) to obtain a two-channel residual network for training.
  • L( ⁇ ) represents the loss function value
  • n represents the number of samples participating in the training in the batch.
  • n is 30
  • x represents the data matrix participating in the training in the batch
  • ⁇ is the summation operator
  • y′ Represents the actual category label matrix corresponding to x
  • log( ⁇ ) represents a logarithm operation
  • y represents a category label matrix of x obtained by network classification.
  • step (3-1) Optimize the dual-channel residual network using the loss function in step (3-1). The training is stopped until the loss function has no new minimum value for 20 consecutive training cycles.
  • Model recognition accuracy (b) Bag-of-Feature (R.Xu, Y.Hirano, R.Tachibana, and S.Kido, "Classification of diffuse lung disease patterns on high-resolution computed tomography by a bag of words approach," in International Conference on Medical Image Computing&Computer-assisted Intervention ( MICCAI), 2011, p.183.) Model recognition accuracy; (c) 5-layer convolutional neural network (CNN-5) (M.Anthimopoulos, S.Christodoulidis, and et.al., "Lung pattern classification for Interstitial lung diseases using a deep convolutional neural network, "IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp.
  • Model identification accuracy (d) Single residual network (ResNet- 18) (Kaiming He, X Jiangyu Zhang, and et.al., “Deep residual learning for image recognition,” in Computer Vision and Pattern Recognition, 2016, pp.770–778.) Model recognition accuracy; (e) is the invention (DB-ResNet- 18) Identify the accuracy rate.
  • Fig. 6 The comparison of the chaotic matrix of the test data with other methods in this embodiment is shown in Fig. 6.
  • (a) is the chaotic matrix of the Bag-of-Feature model;
  • (b) is the 5-layer convolutional nerve.
  • (c) the chaotic matrix of the single residual network (ResNet-18) model;
  • (d) the chaotic matrix of the invention (DB-ResNet-18).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Medical Informatics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法，属于医学图像处理和计算机视觉领域。以217个肺部三维计算机断层扫描(CT)图像为源数据，经过预处理得到若干组数据集合，其中每组数据包含一个CT图像小块、相应的几何信息图像小块和一个类别标签。构建双通道残差网络框架，分别以CT图像小块和相应的几何信息小块为各个通道的输入，通过双通道残差网络分别学习肺部纹理的表观信息和几何信息，并将其有效融合，从而得到较高的识别率。此外，所提出的网络结构清晰，容易构建，易于实现。

Description

一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法 技术领域
本发明属于医学图像处理和计算机视觉领域，涉及利用深度神经网络框架，提取肺部计算机断层扫描(CT)图像的表观和几何信息的相关特征，并对不同类别的肺部CT图像纹理进行分类，具体涉及到一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法。
背景技术
弥漫性肺疾患是在CT图像的肺部区域内，呈现出广泛分布的大面积肺部阴影的肺部疾病的总称。由于这些肺部阴影纹理繁杂，容易混淆，即使对于经验丰富的放射线科专家，也很难做到准确识别不同的肺部纹理。因此，需要建立一种计算机辅助诊断(CAD)系统，对弥漫性肺疾患CT图像的肺部纹理进行准确且高效的自动识别。建立这种CAD系统的一项关键技术，是对CT图像中的肺部区域的任意感兴趣区域(ROI)内的肺部纹理进行准确且高效的自动识别。
传统的CT图像肺部纹理识别，通常基于两步式的方法，即首先以人工方式设计能表征肺部纹理特性的特征量，然后训练能有效区分特征量的分类器。由于分类器训练的技术相对成熟，大多数方法往往采用现有的方法，如神经网络、支持向量机、K-近邻分类器等；因此，科研人员将主要精力放在设计能充分表征肺部纹理特性的特征量上，例如一种基于特征袋的肺部纹理识别方法(R.Xu,Y.Hirano,R.Tachibana,and S.Kido,“Classification of diffuse lung disease patterns on high-resolution computed tomography by a bag of words approach,”in International Conference on Medical Image Computing&Computer-assisted Intervention(MICCAI),2011,p.183.)。
近年来，随着深度学习发展，深度神经网络在图像处理和计算机视觉领域带来了革命性的影响。它将传统的图像识别方法的两步合二为一，即将特征量设计与分类器训练融合成一个整体的端到端框架，直接从图像数据中挖掘和学习能有效表征数据的特征量，并将之应用于图像识别任务中。基于这种模式，深度学习 在自然图像识别、人脸识别等方面，相对于传统的图像识别方法，表现出了卓越的性能。最近，深度学习同样被应用到肺部纹理识别中，例如有研究利用卷积神经网络(CNN)对肺部纹理进行分类(M.Anthimopoulos,S.Christodoulidis,and et.al.,“Lung pattern classification for interstitial lung diseases using a deep convolutional neural network,”IEEE Transactions on Medical Imaging,vol.35,no.5,pp.1207–1216,2016)，使识别性能获得小幅提升。
虽然深度学习方法已经被应用于CT图像的肺部纹理识别中，并且与传统的识别方法相比，在识别精度上有了一定的提升，但是与CAD系统的实际需求相比，肺部纹理识别精度还存在一定的差距。上述基于深度学习的方法存在两个问题，首先，构建的网络层数较少，无法学习高效表征肺部纹理的特征量，不能充分发挥深度学习的优势。其次，针对肺部纹理图像的特殊性，不同的肺部纹理，不仅体现在像素明暗变化所反映的表观区别上，还反映在纹理细微结构的几何差异上。上述基于深度学习的方法仅挖掘肺部纹理的表观信息，忽略了其几何特点，从而未能充分学习有效的特征，无法完成高精度的纹理识别。
基于以上问题，本发明设计了一个基于深度神经网络提取表观和几何特征的肺部纹理识别框架，该框架运用残差网络(Kaiming He,Xiangyu Zhang,and et.al.,“Deep residual learning for image recognition,”in Computer Vision and Pattern Recognition,2016,pp.770–778.)中的残差模块，构建了一个18层的双通道残差网络。它具有双通道的网络结构，能分别提取和学习肺部纹理的表观和几何特征，并将其有机融合，实现了肺部纹理的高精度自动识别。
发明内容
本发明旨在克服现有技术的不足，提供了一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法。该方法具有深层的双通道网络结构，能分别提取肺部纹理的表观和几何信息，并将其有机融合，实现了的肺部纹理的高精度自动识别。
本发明的具体技术方案为，一种基于深度神经网络提取表观和几何特征的肺 部纹理识别方法，包括下列步骤：
1)准备初始数据：初始数据包括用来训练、验证和测试的肺部纹理CT图像小块、对应的几何信息图像小块和对应的类别标签。
2)双通道残差网络的构建：基于残差网络中跳跃结构的思想，构建一个18层的双通道残差网络。
3)基于步骤(2)得到双通道残差网络进行训练。
双通道残差网络的构建，具体包括以下步骤：
2-1)构建的双通道残差网络共18层，由15个卷积层和3个全连接层组成；上面的通道利用(32×32×1的CT图像小块)图像表观信息作为输入，下面的通道利用对应的(32×32×3的几何信息图像小块)图像几何信息作为输入；
2-2)跳跃结构计算卷积层输入信息与经卷积层计算后输出信息的残差，网络通过对残差的学习，可以有效的加深网络层数，避免梯度消失、梯度爆炸问题，使网络能够高效的学习；残差计算公式如下：
y＝F(x，{W i})+x
式中，y表示卷积层的最终输出结果，F表示卷积计算函数，x表示网络的输入，i表示卷积层索引数，定义域为[1，15]，W i表示第i个卷积层的系数，由网络训练得到；
2-3)网络的上、下两通道的最后一个卷积层的输出，在分别经过最大池化后，直接进行串接操作，从而融合上、下通道的信息；
2-4)为了保证双通道残差网络对图片分类时具备旋转不变性和平移不变性，在第一个卷积层后和最后一个卷积层后加入一个最大池化层；
2-5)为了扩大网络的感受野，将第6、第10、第14个卷积层的卷积步长设置为2(默认为1)，起到了2倍下采样的作用，不断扩大网络的感受野。
基于步骤(2)得到的双通道残差网络进行训练，具体包括以下步骤：
3-1)使用小批量训练模式，利用交叉熵的平均值作为损失函数来衡量分类结果与实际类别结果的差别，公式如下：
Figure PCTCN2019070588-appb-000001
式中，L(·)表示损失函数值，n表示该批量中参与训练的样本数，本发明中n为30，x表示该批量中参与训练的数据矩阵，∑为求和运算符，y′表示与x相对应的实际类别标签矩阵，log(·)表示对数运算，y表示经网络分类得到的x的类别标签矩阵。
3-2)使用步骤(3-1)中的损失函数优化双通道残差网络。
本发明的有益效果是：
本发明基于深度学习的思想，利用残差网络中的跳跃结构，构建了一个18层的双通道残差网络。两个通道的输入分别为原始图像和对应的几何信息图像。通过该网络，可以得到较高正确识别率的分类结果。该系统具有以下特点：
1、系统容易构建，仅仅依靠原始的CT图像小块和对应的几何信息图像小块，通过双通道残差网络就可以得到较高正确识别率的分类结果；
2、程序简单，易于实现；
3、利用残差网络中的跳跃结构，在保证网络能够正常学习的情况下，加深了双通道残差网络的层数，提升了网络挖掘图像特征的能力。
4、以残差网络为基础，构建双通道网络，分别学习肺部纹理的表观信息和几何信息，并将其有效融合，显著提升了识别正确率。
附图说明
图1为具体实施流程图。
图2为7类肺部纹理CT图像小块的样例，其中，(a)结节状；(b)肺气肿；(c)蜂窝状；(d)固定型；(e)毛玻璃状；(f)正常；(g)带线条的毛玻璃状。
图3为肺部纹理图，其中(a)CT图像小块；(b)提取的几何信息图小块的样例。
图4为网络结构简图。
图5为分类结果识别正确率与其他方法的比较，其中(a)LeNet-5模型识别准确率；(b)特征袋(Bag-of-Feature)模型识别准确率；(c)5层卷积神经网络(CNN-5)模型识别准确率；(d)单一残差网络(ResNet-18)模型识别准确率；(e)本发明(DB-ResNet-18)识别准确率。
图6为分类结果混乱矩阵与其他方法的比较，其中(a)特征袋(Bag-of-Feature)模型混乱矩阵；(b)5层卷积神经网络(CNN-5)模型混乱矩阵；(c)单一残差网络(ResNet-18)模型混乱矩阵；(d)本发明(DB-ResNet-18)混乱矩阵。
具体实施方式
本发明提出了一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法，结合附图及实施例详细说明如下：
本发明构建了一个双通道残差网络，利用肺部CT图像进行训练，在测试中达到了较高的正确识别率，具体实施流程如图1所示，所述方法包括下列步骤；
1)准备初始数据：
1-1)在实验中共收集了217位病人的肺部CT图像。其中，187位病人的CT图像中包含弥漫性肺病的6种典型纹理，即结节状、肺气肿、蜂窝状、固定型、毛玻璃状和带线条的毛玻璃状纹理；剩余30位病人的CT图像中，只呈现正常的肺部组织纹理。利用此217幅CT图像，生成用于试验的7种肺部纹理图像小块(包含正常组织的纹理)。
1-2)邀请三位富有经验的放射学专家，对217幅CT图像，进行如下操作：
1-2-1)在每幅CT图像的冠状轴方向上选择3张含有典型肺部纹理的断层；
1-2-2)三位专家分别在3个断层上利用标注工具，手动标注出含有典型肺部纹理(包括肺病纹理和正常肺部组织纹理)的区域；
1-2-3)对于每一张断层的标注结果，取三位专家的交集作为最终标记结果。
1-3)利用最终的标注结果，在CT图像的相应断层上，使用32×32×1的扫描框，对CT图像进行采样。从相应断层的左上角开始，分别在水平和垂直方向 上，以8个像素间距为步长进行扫描。当搜索框的中心点落在标记区域内时，保存搜索框内的CT图像区域，并记录好相应CT图像区域的肺部纹理类别，最终得到一系列大小为32×32×1的7类典型肺部纹理CT图像小块。图2给出了所得到的7种肺部纹理CT图像小块的样例。
1-4)对步骤(1-3)所得的全部32×32×1的CT图像区域小块内的全部1024个像素点，按照如下公式计算大小为3×3海森矩阵(Hessian Matrix)。
Figure PCTCN2019070588-appb-000002
式中，H表示海森矩阵，
Figure PCTCN2019070588-appb-000003
表示偏导数计算，x，y，z为三个方向的坐标，其定义域为[0，511]。I表示CT图像的灰度值，其为[0，255]。
对每个像素点所得的海森矩阵，进行矩阵的特征值分解，在每个像素点上将得到3个特征值。将这些特征值按照像素点的位置排列，重组为32×32×3的图像，该图像反映原始的32×32×1的CT图像小块的几何特征，是对应的几何信息图像小块。图3给出了CT图像小块和相应的几何信息图像小块的样例。
1-5)通过步骤(1-3)和(1-4)，共得到72348组数据，每组数据包括一张32×32×1的CT图像小块、32×32×3的几何信息图像小块以及相应肺部纹理的类别标签。随机选取其中的54392组作为训练集和验证集来训练并改善双通道残差网络，剩余的17956组作为测试集对模型和参数已经确定的网络进行测试。训练集、验证集和测试集间互相独立。
2)双通道残差网络的构建：基于残差网络中跳跃结构的思想，构建一个18层的双通道残差网络。图4展示的是网络结构简图。
2-1)本发明构建的双通道残差网络共18层，由15个卷积层和3个全连接层组成。上面的通道利用32×32×1的CT图像小块(图像表观信息)作为输入， 下面的通道利用对应的32×32×3的几何信息图像小块(图像几何信息)作为输入。
2-2)跳跃结构计算卷积层输入信息与经卷积层计算后输出信息的残差，网络通过对残差的学习，可以有效的加深网络层数，避免梯度消失、梯度爆炸问题，使网络能够高效的学习。残差计算公式如下：
y＝F(x，{W i})+x
式中，y表示卷积层的最终输出结果，F表示卷积计算函数，x表示网络的输入，i表示卷积层索引数，定义域为[1，15]，W i表示第i个卷积层的系数，由网络训练得到。
2-3)网络的上、下两通道的最后一个卷积层的输出，在分别经过最大池化后，直接进行串接操作，从而融合上、下通道的信息。
2-4)为了保证双通道残差网络对图片分类时具备旋转不变性和平移不变性，在第一个卷积层后和最后一个卷积层后加入了一个最大池化层。
2-5)为了扩大网络的感受野，将第6、第10、第14个卷积层的卷积步长设置为2(默认为1)，起到了2倍下采样的作用，不断扩大网络的感受野。
3)基于步骤(2)得到双通道残差网络进行训练。
3-1)利用如下公式计算交叉熵的平均值，作为损失函数来衡量分类结果与实际类别结果的差别。
Figure PCTCN2019070588-appb-000004
式中，L(·)表示损失函数值，n表示该批量中参与训练的样本数，本发明中n为30，x表示该批量中参与训练的数据矩阵，∑为求和运算符，y′表示与x相对应的实际类别标签矩阵，log(·)表示对数运算，y表示经网络分类得到的x的类别标签矩阵。
3-2)使用步骤(3-1)中损失函数优化双通道残差网络。直至损失函数在连续的20个训练周期内无新的最小值时，停止训练。
本实施例对测试数据的分类结果识别正确率与其他方法的比较如图5所示，其中(a)为LeNet-5(Y.Lecun,L.Bottou,Y.Bengio,and P.Haffner,“Gradient-based learning applied to document recognition,”Proceedings of the IEEE,vol.86,no.11,pp.2278–2324,1998.)模型识别准确率；(b)为特征袋(Bag-of-Feature)(R.Xu,Y.Hirano,R.Tachibana,and S.Kido,“Classification of diffuse lung disease patterns on high-resolution computed tomography by a bag of words approach,”in International Conference on Medical Image Computing&Computer-assisted Intervention(MICCAI),2011,p.183.)模型识别准确率；(c)为5层卷积神经网络(CNN-5)(M.Anthimopoulos,S.Christodoulidis,and et.al.,“Lung pattern classification for interstitial lung diseases using a deep convolutional neural network,”IEEE Transactions on Medical Imaging,vol.35,no.5,pp.1207–1216,2016.)模型识别准确率；(d)为单一残差网络(ResNet-18)(Kaiming He,Xiangyu Zhang,and et.al.,“Deep residual learning for image recognition,”in Computer Vision and Pattern Recognition,2016,pp.770–778.)模型识别准确率；(e)为本发明(DB-ResNet-18)识别准确率。
本实施例对测试数据的分类结果混乱矩阵与其他方法的比较如图6所示，其中(a)为特征袋(Bag-of-Feature)模型的混乱矩阵；(b)为5层卷积神经网络(CNN-5)模型的混乱矩阵；(c)为单一残差网络(ResNet-18)模型的混乱矩阵；(d)为本发明(DB-ResNet-18)的混乱矩阵。

Claims (2)

  1. 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法，其特征在于，包括下列步骤：
    1)准备初始数据：初始数据包括用来训练、验证和测试的肺部纹理CT图像小块、对应的几何信息图像小块和对应的类别标签；
    2)双通道残差网络的构建：基于残差网络中跳跃结构的思想，构建一个18层的双通道残差网络；
    3)基于步骤2)得到双通道残差网络进行训练。
  2. 根据权利要求1所述的一种基于深度神经网络提取图像表观和几何特征的肺部纹理识别方法，其特征在于，步骤2)中双通道残差网络的构建，具体包括以下步骤：
    2-1)构建的双通道残差网络共18层，由15个卷积层和3个全连接层组成；上面的通道利用图像表观信息作为输入，下面的通道利用对应的图像几何信息作为输入；
    2-2)跳跃结构计算卷积层输入信息与经卷积层计算后输出信息的残差，网络通过对残差的学习；残差计算公式如下：
    y＝F(x，{W i})+x
    式中，y表示卷积层的最终输出结果，F表示卷积计算函数，x表示网络的输入，i表示卷积层索引数，定义域为[1，15]，W i表示第i个卷积层的系数，由网络训练得到；
    2-3)网络的上、下两通道的最后一个卷积层的输出，在分别经过最大池化后，直接进行串接操作，从而融合上、下通道的信息；
    2-4)在第一个卷积层后和最后一个卷积层后加入一个最大池化层；
    2-5)将第6、第10、第14个卷积层的卷积步长设置为2。
PCT/CN2019/070588 2018-03-14 2019-01-07 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法 WO2019174376A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/649,650 US11170502B2 (en) 2018-03-14 2019-01-07 Method based on deep neural network to extract appearance and geometry features for pulmonary textures classification

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810206846.9 2018-03-14
CN201810206846.9A CN108428229B (zh) 2018-03-14 2018-03-14 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法

Publications (1)

Publication Number Publication Date
WO2019174376A1 true WO2019174376A1 (zh) 2019-09-19

Family

ID=63157963

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/070588 WO2019174376A1 (zh) 2018-03-14 2019-01-07 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法

Country Status (3)

Country Link
US (1) US11170502B2 (zh)
CN (1) CN108428229B (zh)
WO (1) WO2019174376A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111612739A (zh) * 2020-04-16 2020-09-01 杭州电子科技大学 一种基于深度学习的脑梗死分类方法
CN111985532A (zh) * 2020-07-10 2020-11-24 西安理工大学 一种场景级上下文感知的情感识别深度网络方法

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108428229B (zh) * 2018-03-14 2020-06-16 大连理工大学 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法
CN109300136B (zh) * 2018-08-28 2021-08-31 众安信息技术服务有限公司 一种基于卷积神经网络的危及器官自动分割方法
CN109460756B (zh) * 2018-11-09 2021-08-13 天津新开心生活科技有限公司 医学影像处理方法、装置、电子设备及计算机可读介质
CN109871806A (zh) * 2019-02-21 2019-06-11 山东大学 基于深度残差纹理网络的地形识别方法及系统
CN110321920B (zh) * 2019-05-08 2021-10-22 腾讯科技（深圳）有限公司 图像分类方法、装置、计算机可读存储介质和计算机设备
CN110472663A (zh) * 2019-07-15 2019-11-19 西北工业大学 基于内省学习的遥感图像分类方法
CN111652170A (zh) * 2020-06-09 2020-09-11 电子科技大学 基于二通道残差深度神经网络的二次雷达信号处理方法
CN111739076B (zh) * 2020-06-15 2022-09-30 大连理工大学 面向多种ct肺部纹理识别的非监督保内容域适应方法
CN111739075B (zh) * 2020-06-15 2024-02-06 大连理工大学 一种结合多尺度注意力的深层网络肺部纹理识别方法
CN112017136A (zh) * 2020-08-06 2020-12-01 杭州深睿博联科技有限公司 一种基于深度学习的肺部ct图像参数重建方法、系统、终端及存储介质
CN112270667B (zh) * 2020-11-02 2023-04-25 郑州大学 一种基于ti-rads的集成深度学习多标签识别方法
CN112465830B (zh) * 2020-11-11 2024-04-26 上海健康医学院 一种磨玻璃样肺结节自动分割方法及计算机设备
CN112488234B (zh) * 2020-12-10 2022-04-29 武汉大学 一种基于注意力池化的端到端组织病理图像分类方法
CN112560932A (zh) * 2020-12-10 2021-03-26 山东建筑大学 一种基于双分支网络特征融合的车辆重识别方法
CN113128601B (zh) * 2021-04-22 2022-04-29 北京百度网讯科技有限公司 分类模型的训练方法和对图像进行分类的方法
CN113065570B (zh) * 2021-06-03 2021-08-24 北京远鉴信息技术有限公司 一种图像审核的检测方法、装置及电子设备
CN113269765B (zh) * 2021-06-04 2022-10-28 重庆大学 可扩张卷积神经网络训练方法及ct影像分割模型构建方法
CN113469972B (zh) * 2021-06-30 2024-04-23 沈阳东软智能医疗科技研究院有限公司 标注医学切片图像的方法、装置、存储介质及电子设备
CN113657486B (zh) * 2021-08-16 2023-11-07 浙江新再灵科技股份有限公司 基于电梯图片数据的多标签多属性分类模型建立方法
CN113837199B (zh) * 2021-08-30 2024-01-09 武汉理工大学 一种基于跨层残差双路金字塔网络的图像特征提取方法
CN113888515A (zh) * 2021-10-12 2022-01-04 浙江大学 基于深度学习和人类视觉特性的双通道立体图像质量评价方法
CN114359157B (zh) * 2021-12-08 2023-06-16 南通大学 一种基于深度学习的结直肠息肉检测方法
CN114708236B (zh) * 2022-04-11 2023-04-07 徐州医科大学 一种基于tsn和ssn在超声图像中的甲状腺结节良恶性分类方法
CN114648680B (zh) * 2022-05-17 2022-08-16 腾讯科技（深圳）有限公司 图像识别模型的训练方法、装置、设备、介质
CN115272218A (zh) * 2022-07-22 2022-11-01 重庆文理学院 一种基于cbam机制的残差网络的医学影像辅助检测方法
CN116089883B (zh) * 2023-01-30 2023-12-19 北京邮电大学 用于提高已有类别增量学习新旧类别区分度的训练方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106530296A (zh) * 2016-11-07 2017-03-22 首都医科大学 一种基于pet/ct图像特征的肺部检测方法和装置
CN106650830A (zh) * 2017-01-06 2017-05-10 西北工业大学 基于深层模型和浅层模型决策融合的肺结节ct图像自动分类方法
CN107016665A (zh) * 2017-02-16 2017-08-04 浙江大学 一种基于深度卷积神经网络的ct肺结节检测方法
CN108428229A (zh) * 2018-03-14 2018-08-21 大连理工大学 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5873824A (en) * 1996-11-29 1999-02-23 Arch Development Corporation Apparatus and method for computerized analysis of interstitial infiltrates in chest images using artificial neural networks
US20050207630A1 (en) * 2002-02-15 2005-09-22 The Regents Of The University Of Michigan Technology Management Office Lung nodule detection and classification
US6819790B2 (en) * 2002-04-12 2004-11-16 The University Of Chicago Massive training artificial neural network (MTANN) for detecting abnormalities in medical images
US10593035B2 (en) * 2015-03-18 2020-03-17 University Of South Florida Image-based automated measurement model to predict pelvic organ prolapse
US10458895B2 (en) * 2016-06-06 2019-10-29 Case Western Reserve University Predicting response to pemetrexed chemotherapy in non-small cell lung cancer (NSCLC) with baseline computed tomography (CT) shape and texture features
US10970887B2 (en) * 2016-06-24 2021-04-06 Rensselaer Polytechnic Institute Tomographic image reconstruction via machine learning
CN106971198A (zh) * 2017-03-03 2017-07-21 北京市计算中心 一种基于深度学习的尘肺病等级判定方法及系统
CN107590797B (zh) * 2017-07-26 2020-10-30 浙江工业大学 一种基于三维残差神经网络的ct影像肺结节检测方法
CN107644419A (zh) * 2017-09-30 2018-01-30 百度在线网络技术（北京）有限公司 用于分析医学影像的方法和装置
US10586132B2 (en) * 2018-01-08 2020-03-10 Visteon Global Technologies, Inc. Map and environment based activation of neural networks for highly automated driving
US10970837B2 (en) * 2019-03-18 2021-04-06 Siemens Healthcare Gmbh Automated uncertainty estimation of lesion segmentation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106530296A (zh) * 2016-11-07 2017-03-22 首都医科大学 一种基于pet/ct图像特征的肺部检测方法和装置
CN106650830A (zh) * 2017-01-06 2017-05-10 西北工业大学 基于深层模型和浅层模型决策融合的肺结节ct图像自动分类方法
CN107016665A (zh) * 2017-02-16 2017-08-04 浙江大学 一种基于深度卷积神经网络的ct肺结节检测方法
CN108428229A (zh) * 2018-03-14 2018-08-21 大连理工大学 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111612739A (zh) * 2020-04-16 2020-09-01 杭州电子科技大学 一种基于深度学习的脑梗死分类方法
CN111612739B (zh) * 2020-04-16 2023-11-03 杭州电子科技大学 一种基于深度学习的脑梗死分类方法
CN111985532A (zh) * 2020-07-10 2020-11-24 西安理工大学 一种场景级上下文感知的情感识别深度网络方法
CN111985532B (zh) * 2020-07-10 2021-11-09 西安理工大学 一种场景级上下文感知的情感识别深度网络方法

Also Published As

Publication number Publication date
CN108428229A (zh) 2018-08-21
US20200258218A1 (en) 2020-08-13
CN108428229B (zh) 2020-06-16
US11170502B2 (en) 2021-11-09

Similar Documents

Publication Publication Date Title
WO2019174376A1 (zh) 一种基于深度神经网络提取表观和几何特征的肺部纹理识别方法
CN105069746B (zh) 基于局部仿射和颜色迁移技术的视频实时人脸替换方法及其系统
CN111476292B (zh) 医学图像分类处理人工智能的小样本元学习训练方法
CN110059697A (zh) 一种基于深度学习的肺结节自动分割方法
CN105574527A (zh) 一种基于局部特征学习的快速物体检测方法
Iqtait et al. Feature extraction for face recognition via active shape model (ASM) and active appearance model (AAM)
CN104008375B (zh) 基于特征融合的集成人脸识别方法
CN106951840A (zh) 一种人脸特征点检测方法
CN111126240B (zh) 一种三通道特征融合人脸识别方法
CN105760874A (zh) 面向尘肺的ct图像处理系统及其ct图像处理方法
CN104484886B (zh) 一种mr图像的分割方法及装置
CN108549912A (zh) 一种基于机器学习的医学图像肺结节检测方法
CN110675421B (zh) 基于少量标注框的深度图像协同分割方法
CN111767952A (zh) 一种可解释的肺结节良恶性分类方法
CN103839074B (zh) 一种基于素描线段信息和空间金字塔匹配的图像分类方法
CN116579982A (zh) 一种肺炎ct图像分割方法、装置及设备
CN108491863A (zh) 基于非负矩阵分解和卷积神经网络的彩色图像处理方法
CN113838009A (zh) 一种基于半监督机制的异常细胞检测假阳性抑制方法
CN107392251A (zh) 一种使用分类图片提升目标检测网络性能的方法
CN105138975A (zh) 一种基于深度信念网络的人体肤色区域分割方法
CN107330948B (zh) 一种基于流行学习算法的fMRI数据二维可视化方法
CN109410233A (zh) 一种边缘特征约束的高分辨率图像道路精准提取方法
CN108898601A (zh) 基于随机森林的股骨头图像分割装置及分割方法
Zhang et al. Simultaneous lung field detection and segmentation for pediatric chest radiographs
CN117036288A (zh) 一种面向全切片病理图像的肿瘤亚型诊断方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19767636

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19767636

Country of ref document: EP

Kind code of ref document: A1