WO2022228396A1 - 内窥镜多光谱图像处理系统及处理和训练方法 - Google Patents

内窥镜多光谱图像处理系统及处理和训练方法 Download PDF

Info

Publication number
WO2022228396A1
WO2022228396A1 PCT/CN2022/089022 CN2022089022W WO2022228396A1 WO 2022228396 A1 WO2022228396 A1 WO 2022228396A1 CN 2022089022 W CN2022089022 W CN 2022089022W WO 2022228396 A1 WO2022228396 A1 WO 2022228396A1
Authority
WO
WIPO (PCT)
Prior art keywords
endoscope
image processing
multispectral image
processing system
output
Prior art date
Application number
PCT/CN2022/089022
Other languages
English (en)
French (fr)
Inventor
杜武华
钱大宏
徐健玮
黄显峰
Original Assignee
山东威高宏瑞医学科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 山东威高宏瑞医学科技有限公司 filed Critical 山东威高宏瑞医学科技有限公司
Publication of WO2022228396A1 publication Critical patent/WO2022228396A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0075—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition

Definitions

  • the present application relates to the field of medical technology, and in particular, to an endoscope multispectral image processing technology.
  • Medical endoscopes have become more and more popular in clinical applications, and have become one of the important tools for human observation and diagnosis of internal organ lesions.
  • Medical endoscope is currently a widely used medical device. Medical endoscope is an effective, non-invasive and low-cost medical diagnostic tool that can display images of diseased tissue in the human body. Studies have shown that gastrointestinal endoscopy, the gold standard for gastrointestinal tumor screening, can reduce the risk of death from gastrointestinal tumors by detecting tumors at an early stage and removing precancerous lesions.
  • wavelengths of narrow-band light can improve the contrast between the lesion and surrounding tissues in clinical detection
  • the NBI imaging technology designed by Olympus Corporation of Japan selected narrow-band light of 415nm and 540nm.
  • Fujifilm Corporation of Japan The developed blue laser imaging technology BLI (Blue Laser Imaging) uses a semiconductor laser as a light source, and the narrow-band light mode contains two wavelengths of 410nm and 450nm.
  • BLI Blue Laser Imaging
  • the wavelength band of 550-700 nm has also been proved to improve the clinical images of some diseases.
  • the multispectral system covers most of the narrow-band light in the visible light band, the narrow-band light of some wavelengths may not be significantly helpful for the prediction results of the neural network or the observation of the doctor, and the narrow-band light of other wavelengths (such as NBI technology).
  • the chosen 415nm and 540nm) may be more helpful in predicting outcomes and physician observations. This may be because the features contained in the pictures under different wavelengths of light may be of different importance.
  • NBI imaging technology uses narrow-band light with two wavelengths of 415nm and 540nm, because the larger blood vessels under the mucosa can absorb blue light and display blue-green, The microvessels can absorb blue-green and appear brown, thereby improving the contrast between the microvessels and the surrounding tissue. Therefore, NBI technology believes that the characteristics of the pictures under the two wavelengths of 415nm and 540nm are more important and more conducive to the doctor's observation.
  • the purpose of the present application is to provide an endoscope multispectral image processing system and a training and processing method, which makes it easier for doctors to see the lesions.
  • the present application discloses an endoscope multispectral image processing system, including:
  • an encoder configured to encode the n endoscopic images obtained under the input n wavelengths, respectively, and output n feature maps corresponding to n different wavelengths, where n is an integer greater than 1;
  • the wavelength selection module is configured to output respective weight matrices according to the inputted n feature maps, the weight matrices represent the importance of the feature maps of different wavelengths, and the weight matrix is used for the input splicing module feature maps. control;
  • the splicing module is configured to splicing the feature maps output by the encoder to obtain a splicing feature map
  • a classifier configured to classify the stitched feature map, and output a classification result.
  • a feature map processing module is further included, which is configured to filter and/or weight the n feature maps output by the encoder according to the weight matrix and output to the splicing module.
  • the weight matrix includes n weights, respectively corresponding to the feature maps of n different wavelengths;
  • the screening includes screening out the feature maps corresponding to the wavelengths corresponding to the weights smaller than the preset threshold among the n weights of the weight matrix;
  • the weighting process includes multiplying the feature maps by corresponding weights output by the wavelength selection module, respectively.
  • the weight matrix used to control the feature map input to the splicing module further includes:
  • the preprocessing module is configured to filter and/or weight the n endoscopic images obtained under the n wavelengths input to the encoder according to the weight matrix.
  • an imaging device is also included for capturing an endoscopic image
  • the imaging device is configured to control the imaging wavelength used or received to capture the endoscopic image according to the weight matrix.
  • the control includes screening and/or light intensity treatment
  • the weight matrix includes n weights, respectively corresponding to the feature maps of n different wavelengths;
  • the screening includes screening out the light of the wavelength corresponding to the weight less than the preset threshold among the n weights;
  • the light intensity processing includes controlling the intensity of each wavelength of light according to the n weights.
  • the system uses multiple sets of training data for supervised training, each set of training data includes n endoscopic images obtained under n different wavelengths and pre-calibrated images based on the n endoscopic images. Classification results.
  • each encoder is configured to encode an inputted endoscopic image obtained at one wavelength to obtain a feature map corresponding to the wavelength, and there are n different wavelengths in total
  • the endoscopic images obtained below are respectively input to the n encoders, and n feature maps corresponding to n different wavelengths are output.
  • the n encoders are n deep neural networks; the classifier is a deep neural network; and the wavelength selection module is a self-attention-based deep neural network.
  • the application also discloses an endoscope multispectral image processing method comprising:
  • n endoscopic images obtained under n different wavelengths are respectively input into the encoder of the endoscopic multispectral image processing system as described above, and the output of the wavelength selection module of the system corresponding to the n different wavelengths is obtained.
  • An endoscopic image corresponding to a wavelength corresponding to the weight greater than the preset threshold is output.
  • the application also discloses a training method for an endoscope multispectral image processing system, comprising:
  • each set of training data includes n endoscopic images obtained under n different wavelengths and pre-calibrated classification results based on the n endoscopic images;
  • the endoscopic multispectral image processing system as described above is used for training using the multiple sets of data as a whole, wherein, for each set of training data, the n endoscopic images are used as the encoder of the system. input, and use the classification result as the classifier output of the system.
  • the application also discloses an endoscope multispectral image processing method comprising:
  • the output result of the classifier of the endoscope multispectral image processing system is acquired and displayed on the display.
  • the neural network in the system automatically ranks the importance of each wavelength of light, and then feeds it back to the system.
  • the system automatically selects the appropriate wavelength and light intensity for imaging again, and re-enters the new image into the next step.
  • the classification network performs pathological classification, and adjusts and outputs new images for the doctor to observe, so that the doctor can easily see the lesions.
  • the system of the present application sets all the images corresponding to the screened out wavelengths as black.
  • the input of the neural network of the system of the present application is the image corresponding to the wavelength filtered by the above network, that is, the image corresponding to the wavelength greater than the threshold is retained, and the image corresponding to the wavelength less than the threshold is set to black, and the classification result is output.
  • supervised training is performed, and each set of training data includes m images under m wavelengths greater than the threshold and n-m black images less than the threshold, as well as pre-calibrated classification results.
  • the image selection module adjusts the light source according to the weights corresponding to the n wavelengths.
  • the light intensity of the wavelength with the weight wi smaller than the threshold t is set to 0, and the light intensity ratio of the wavelength with the weight wi greater than the threshold t is the same as the weight ratio. Thereby, the noise wavelengths are removed, the image is clearer, and the shooting efficiency is improved.
  • features A+B+C are disclosed, and in another example, features A+B+D+E are disclosed, and features C and D are equivalent technical means that serve the same function. It can be used as soon as it is used, it is impossible to use it at the same time, and feature E can technically be combined with feature C, then the solution of A+B+C+D should not be regarded as having been recorded because it is technically infeasible, while A+B+ The C+E scheme shall be deemed to have been documented.
  • FIG. 1 is a schematic structural diagram of an endoscope multispectral image processing system according to the present application.
  • FIG. 2 is a schematic structural diagram of an endoscope multispectral image processing system according to an embodiment of the present application
  • FIG. 3 is a schematic structural diagram of obtaining and classifying a new image by adjusting the wavelength of a light source according to a characteristic matrix according to an embodiment of an endoscope multispectral image processing system of the present application;
  • FIG. 4 is a schematic structural diagram of a wavelength selection module based on CNN+XGBoost of the endoscope multispectral image processing system of the present application;
  • FIG. 5 is a schematic structural diagram of a classifier according to an embodiment of the CNN+XGBoost-based wavelength selection module of the endoscope multispectral image processing system of the present application;
  • FIG. 6 is a schematic structural diagram of an OOB-based wavelength selection module of the endoscope multispectral image processing system according to the present application.
  • FIG. 7 is a schematic flowchart of an endoscope multispectral image processing method according to an embodiment of the present application.
  • FIG. 8 is a schematic diagram of the use flow of the endoscope multispectral image processing system according to the present application.
  • FIG. 9 is a schematic diagram of a polyp real-time target detection model of the endoscopic multispectral image processing system according to the present application.
  • the first embodiment of the present application relates to an endoscope multispectral image processing system.
  • the endoscope multispectral image processing system includes: an encoder, a wavelength selection module, a splicing module, and a classifier.
  • the encoder is configured to respectively encode the n endoscopic images obtained under the input n wavelengths, and output n feature maps corresponding to the n different wavelengths, where n is an integer greater than 1.
  • n encoders There are n encoders, and each encoder is configured to encode an inputted endoscopic image obtained at one wavelength, and obtain a feature map corresponding to the wavelength. There are a total of n endoscopic images obtained at different wavelengths, respectively. Input n encoders, and output n feature maps corresponding to n different wavelengths.
  • the system of the present application is used to collect images under the illumination of narrow-band light of different wavelengths in the same area, and the dimension of the spectral wavelength is added on the basis of the two-dimensional image, which becomes a data cube (Data Cube) or ultra-high light. Cubes (Hypercubes).
  • the multispectral spectrum includes visible light with wavelengths between 400nm and 700nm. Assuming that a central wavelength is taken every 10nm for imaging, 31 images of the same area under light of different wavelengths can be obtained, and then multispectral images with a spectral dimension of 31 can be obtained.
  • Spectral image. In this embodiment, n 31. In other embodiments, n may also be other values, such as 5, 15, and so on.
  • the pictures L1 ⁇ Ln under each wavelength in the data cube are respectively input to the respective encoders 1 ⁇ n for feature extraction.
  • These encoders are composed of deep neural networks, including convolutional layers, downsampling layers, pooling layers, batch reduction One layer, etc. These encoders can share all or part of the parameters, and the corresponding feature maps 1 to n are obtained after passing through the encoders.
  • the wavelength selection module is configured to output a weight matrix according to the input n feature maps.
  • the weight matrix represents the importance of the feature maps of different wavelengths. The larger the weight, the greater the contribution of the wavelength to the network prediction; It means that the wavelength contributes less to the system prediction.
  • the weight matrix is used to control the feature map of the input stitching module.
  • the wavelength selection module can be implemented in a variety of ways.
  • a neural network based on a self-attention mechanism is used to implement the wavelength selection module.
  • the wavelength selection module splices these feature maps in the channel dimension.
  • the spliced feature maps go through several fully connected layers and activation layers, and finally normalize to 0 through a Softmax or Sigmoid activation function.
  • ⁇ 1 interval generate a weight matrix W containing n elements.
  • the weight coefficients w1 to wn in the weight matrix represent the importance of pictures at wavelengths L1 to Ln, and w1 to wn are all between 0 to 1.
  • the wavelength selection module obtains a weight matrix based on CNN+XGBoost, as shown in FIG. 4 .
  • a deep neural network-based model is used to extract feature vectors.
  • the network includes an encoder, a global pooling layer (Global Average Pooling, GAP), and a classifier.
  • GAP Global Average Pooling
  • each set of training data includes n endoscopic images obtained under n different wavelengths and pre-calibrated classification results based on the n endoscopic images.
  • the picture under each wavelength obtains a feature map 1-n of w*h*c through encoders 1-n, where w, h are the width and height of the feature map, and c is the number of channels of the feature map.
  • each feature map gets a 1*c feature vector. That is, the image at each wavelength corresponds to a 1*c eigenvector, the total length of the eigenvector is n*c, and n is the number of wavelengths.
  • each set of training data includes the feature vectors obtained by the deep neural network model of n endoscopic images obtained under n different wavelengths and the pre-calibrated classification results based on the n endoscopic images.
  • the wavelength selection module sorts the wavelengths based on OOB, and the training process and steps of the sorting further include:
  • the neural network model is trained, and then the OOB (Out of Bag) data is used to calculate the feature importance.
  • the Out of Bag (test) data is used to quantify the feature importance.
  • first use the trained model to score the OOB data, and calculate the accuracy or AUC (Area Under Curve) or other defined evaluation indicators; then perform the following operations on the images under each wavelength in the OOB data:
  • each set of training data includes n endoscopic images obtained at n different wavelengths and pre-calibrated classification results based on the n endoscopic images;
  • the index change rate ⁇ that is, the difference between the evaluation index of the original test set data and the evaluation index after setting the image under a certain wavelength as random noise.
  • Using the weight matrix to control the feature map of the input stitching module can be implemented in various ways.
  • FIG. 3 it also includes a feature map processing module, which is configured to screen and/or weight the n feature maps output by the encoder according to the weight matrix and output to the splicing module. .
  • the weight matrix includes n weights, respectively corresponding to the feature maps of n different wavelengths.
  • the screening includes screening out feature maps corresponding to wavelengths corresponding to weights smaller than a preset threshold among the n weights of the weight matrix.
  • a preset threshold t is set. If wi ⁇ t, then the spectral picture Li of the wavelength corresponding to this wi is considered to have no significant effect on network prediction. Then, the wavelengths (wi ⁇ t) that have no significant effect on network prediction can be filtered out through this weight matrix W.
  • the weighting process includes multiplying the feature maps by the corresponding weights output by the wavelength selection module, respectively. Input the pictures L1 ⁇ Ln under each wavelength in the data cube to the respective encoders 1 ⁇ n for feature extraction, and then multiply the feature maps 1 ⁇ n by the weight coefficients w1 ⁇ wn respectively, and splicing in the channel dimension to realize the feature Fusion, and then sent to the classifier for classification to obtain the final pathological classification result.
  • the above-mentioned screening process is introduced. Since the number of wavelengths after screening is changed from n to m, and there is no corresponding picture for some wavelengths Li, the input pixels corresponding to the wavelengths removed by the screening are all set to (0,0 ,0), which is black.
  • these encoders are composed of deep neural networks, including convolutional layers, downsampling layers, pooling layers, batch normalization layers, etc. These encoders can share all or part of the parameters, and the corresponding feature maps 1 to n are obtained after passing through the encoders.
  • a preprocessing module is also included, configured to perform screening and/or weighting processing on the n endoscopic images obtained under the n wavelengths input to the encoder according to the weight matrix.
  • the weight matrix includes n weights, respectively corresponding to the feature maps of n different wavelengths.
  • Screening includes screening out images of wavelengths corresponding to weights smaller than a preset threshold among the n weights, or setting light intensities corresponding to wavelengths with weights smaller than a threshold to zero in the received endoscopic image.
  • an imaging device is further included for capturing an endoscopic image; the imaging device is configured to control the imaging wavelength used or received for capturing an endoscopic image according to a weight matrix, including filtering and/or intensity processing to obtain a new multispectral image, which is then re-input to the encoder, weighted, and then input to the stitching module.
  • the weight matrix includes n weights, respectively corresponding to the feature maps of n different wavelengths.
  • the screening includes screening out light of wavelengths corresponding to the weights smaller than the preset threshold among the n weights.
  • the light source does not use the light of the wavelength with such a small weight for photographing, or the image receiving device does not accept the light of the wavelength with such a small weight.
  • Light intensity processing includes controlling the intensity of each wavelength of light according to n weights.
  • the splicing module is configured to splicing the feature maps output by the encoder to obtain the splicing feature maps. These feature maps are spliced in the channel dimension, and the spliced feature maps are sent to the classifier to obtain the final pathological classification result.
  • the classifier configured to classify the concatenated feature map, and output the classification result.
  • the classifier may be composed of a deep neural network, including a convolutional layer, a downsampling layer, a pooling layer, a batch normalization layer, a fully connected layer, a Softmax/Sigmoid activation layer, and the like.
  • the n encoders are n deep neural networks; the classifiers are deep neural networks; and the wavelength selection module is a self-attention-based deep neural network.
  • the system uses multiple sets of training data for supervised training, and each set of training data includes n endoscopic images obtained under n different wavelengths and pre-calibrated classification results based on the n endoscopic images.
  • the second embodiment of the present application relates to an endoscope multispectral image processing method, as shown in FIG. 7 , including:
  • step 701 n endoscopic images obtained under n different wavelengths are acquired.
  • the n endoscopic images can be acquired in various ways.
  • step 701 further includes switching n types of illumination light, respectively acquiring n endoscopic images under each illumination light, the center wavelength of each illumination light is different, and each illumination light is different.
  • the wavelength range is less than a predetermined threshold.
  • step 701 further includes switching the filter device in front of the camera for capturing endoscopic images n times to acquire n endoscopic images, and only one filter device is switched each time.
  • Light of a predetermined wavelength range can enter the camera, and the wavelength range that can enter the camera is different after each switching of the filter device.
  • step 702 input the n endoscopic images obtained under n different wavelengths into the encoder of the endoscopic multispectral image processing system as in the first embodiment, and obtain the output of the wavelength selection module of the system corresponding to n n weights for different wavelengths.
  • step 703 select a weight greater than a preset threshold from the n weights.
  • step 704 output the endoscopic image corresponding to the wavelength corresponding to the weight greater than the preset threshold.
  • the image input to the encoder can come from a video, and the wavelengths corresponding to the images in the video are switched in turn under n different wavelengths, and one of each consecutive n endoscopic images will be selected for output, and the above steps are continuously performed. Therefore, the output endoscopic images also constitute a continuous video.
  • the output image or video can be displayed on the monitor.
  • this embodiment is an image processing method embodiment that cooperates with the first embodiment, the technical details in the first embodiment can be applied to this embodiment, and the technical details in this embodiment can also be applied to the first embodiment. an embodiment.
  • the third embodiment of the present application relates to a training method for an endoscope multispectral image processing system, including:
  • each set of training data includes n endoscopic images obtained under n different wavelengths and pre-calibrated classification results based on the n endoscopic images.
  • the endoscopic multispectral image processing system is used for training using multiple sets of data as a whole, wherein, for each set of training data, n endoscopic images are used as the input of the encoder of the system, and The classification result is output as the classifier of the system.
  • this embodiment is a training method embodiment that cooperates with the first embodiment, the technical details in the first embodiment can be applied to this embodiment, and the technical details in this embodiment can also be applied to the first embodiment. implementation.
  • the fourth embodiment of the present application relates to an endoscope multispectral image processing method, including:
  • the n endoscopic images obtained at n different wavelengths are respectively input to the n encoders of the endoscopic multispectral image processing system according to the first embodiment.
  • this embodiment is an image processing method embodiment that cooperates with the first embodiment, the technical details in the first embodiment can be applied to this embodiment, and the technical details in this embodiment can also be applied to the first embodiment. an embodiment.
  • the CAD system is the system described in the first embodiment of the present application, and the system mainly includes the following steps:
  • Figure 9 is a polyp real-time object detection model.
  • the image x under white light is used as input, and an encoder is input for feature extraction, and the feature map obtained after feature extraction is regressed through a detector to obtain the coordinate y of the location of the polyp.
  • the encoder and detector are composed of deep neural networks, including convolutional layers, downsampling layers, upsampling layers, pooling layers, batch normalization layers, activation layers, etc.
  • the system will prompt the doctor with a red box on the monitor where the polyp is located, and sound an alarm.
  • the image processing center feeds back to the light source, switches to the multi-spectral mode, and obtains multiple wavelengths of continuous narrow-band light in the same area. Multispectral image below.
  • the system analyzes multispectral images in real time, and judges the pathological properties by establishing a good spectral feature ranking and interpretability model.
  • the most suitable narrow-band light of single or multiple wavelengths and the intensity of each wavelength light are automatically selected, and are fed back to the light source through the image processing center for switching.
  • an action is performed according to a certain element, it means at least that the action is performed according to the element, which includes two situations: the action is performed only according to the element, and the action is performed according to the element and Other elements perform this behavior.
  • Expressions such as multiple, multiple, multiple, etc. include 2, 2, 2, and 2 or more, 2 or more, and 2 or more.

Abstract

本申请涉及医疗技术，公开了一种内窥镜多光谱图像处理系统及处理和训练方法，可以选择基于深度学习特征排序对波长进行筛选，使得医生更容易看清病灶。该系统包括：编码器，被配置为对输入的n种波长下获得的n个内窥镜图像分别进行编码，输出对应于n种不同波长的n个特征图，其中n为大于1的整数；波长选择模块，被配置为根据输入的所述n个特征图，输出分别权重矩阵，所述权重矩阵代表不同波长的特征图的重要性，所述权重矩阵用于对输入拼接模块的特征图进行控制；所述拼接模块，被配置为对所述编码器输出的特征图进行拼接，得到拼接特征图；分类器，被配置为对所述拼接特征图进行分类，输出分类结果。

Description

内窥镜多光谱图像处理系统及处理和训练方法 技术领域
本申请涉及医疗技术领域，特别涉及一种内窥镜多光谱图像处理技术。
背景技术
随着现代科学技术的发展，医用内窥镜在临床上的应用越来越普及，已经成为人类观察、诊断人体内器官病变的重要工具之一。医用内窥镜是当前应用非常广泛的医疗器械，医用内窥镜是一种有效的、非入侵式与低成本的医学诊断工具，可以显示人体内病变组织的图像。研究表明，作为消化道肿瘤筛查的金标准，消化道内窥镜检查可以通过在早期发现肿瘤并切除癌前病变来降低因消化道肿瘤死亡的风险。
普通电子内窥镜使用白光作为照明光源，虽然能提供清晰的消化道黏膜组织图像并且可以发现形态或颜色明显变化的病灶，但是对于消化道内微小而扁平的早期癌变和各级血管则难以观察，也无法提高微细结构的对比度，容易发生漏诊和误诊。为了克服这些缺点，应用多光谱成像技术的内窥镜应运而生。主要包括奥林巴斯公司研制的窄带光成像技术NBI(Narrow Band Imaging)、自体荧光成像技术AFI(Auto Fluorescence Imaging)、日本宾得公司发明的新型图像增强I-Scan技术、日本富士能公司开发的智能分光比色技术FICE(Fuji Intelligent Chromo Endoscopy)以及日本富士胶片公司研发的蓝激光成像技术BLI(Blue Laser Imaging)。
尽管不同研究提出某些波长的窄带光可以在临床检测中改善病灶与周围组织之间的对比度，比如日本奥林巴斯公司设计的NBI成像技术选 择了415nm、540nm的窄带光，日本富士胶片公司研发的蓝激光成像技术BLI(Blue Laser Imaging)使用半导体激光器作为光源，其中窄带光模式包含了410nm和450nm两种波长。另外一些研究表明，550～700nm的波段也被证明对部分疾病的临床图像有改善作用。但是具体哪些波长的窄带光对图像的改善效果最好，以及各个波长窄带光的强度的选择，并没有达成一个统一的共识。
并且，尽管多光谱系统涵盖了可见光波段中的大部分窄带光，但部分波长的窄带光可能对神经网络的预测结果或者医生的观察并没有显著性帮助，另外部分波长的窄带光(如NBI技术选择的415nm和540nm)可能对预测结果和医生的观察帮助更大。这可能是由于不同波长光下的图片包含的特征可能重要性不一样，比如NBI成像技术使用了415nm和540nm两个波长的窄带光，因为黏膜下较大的血管可以吸收蓝光而显示蓝绿色，而微血管可以吸收蓝绿色而显示棕色，从而提高了微血管与周围组织的对比度。因此NBI技术认为415nm和540nm两个波长光下的图片的特征更为重要，更有利于医生观察。
因此，如何自动选择合适的波长使得医生更容易看清病灶是本领域非常重要的问题。
发明内容
本申请的目的在于提供一种内窥镜多光谱图像处理系统及训练和处理方法，使得医生更容易看清病灶。
本申请公开了一种内窥镜多光谱图像处理系统，包括：
编码器，被配置为对输入的n种波长下获得的n个内窥镜图像分别进行编码，输出对应于n种不同波长的n个特征图，其中n为大于1的整 数；
波长选择模块，被配置为根据输入的所述n个特征图，输出分别权重矩阵，所述权重矩阵代表不同波长的特征图的重要性，所述权重矩阵用于对输入拼接模块的特征图进行控制；
所述拼接模块，被配置为对所述编码器输出的特征图进行拼接，得到拼接特征图；
分类器，被配置为对所述拼接特征图进行分类，输出分类结果。
在一个优选例中，还包括特征图处理模块，被配置为根据所述权重矩阵对所述编码器输出的n个特征图进行筛选和/或加权处理后输出给所述拼接模块。
在一个优选例中，
所述权重矩阵包括n个权重，分别对应n种不同波长的特征图；
所述筛选包括筛除所述权重矩阵的n个权重中小于预设阈值的权重对应的波长对应的所述特征图；
所述加权处理包括将所述特征图分别乘以所述波长选择模块输出的对应权重。
在一个优选例中，所述权重矩阵用于对输入所述拼接模块的特征图进行控制进一步包括：
预处理模块，被配置为根据所述权重矩阵对输入到所述编码器的n种波长下获得的n个内窥镜图像进行筛选和\或加权处理。
在一个优选例中，还包括成像装置，用于拍摄内窥镜图像；
所述成像装置被配置为根据所述权重矩阵对拍摄内窥镜图像所使用或接收的成像波长进行控制。
在一个优选例中，
所述控制包括筛选和/或光强处理；
所述权重矩阵包括n个权重，分别对应n种不同波长的特征图；
所述筛选包括筛除所述n个权重中小于预设阈值的权重对应的波长的光；
所述光强处理包括根据所述n个权重控制每个波长的光的强度。
在一个优选例中，所述系统使用多组训练数据进行有监督方式训练，每组训练数据包括n种不同波长下获得的n个内窥镜图像和基于该n个内窥镜图像预先标定的分类结果。
在一个优选例中，所述编码器有n个，每个编码器被配置为对输入的一种波长下获得的内窥镜图像进行编码，得到对应该波长的特征图，共有n种不同波长下获得的内窥镜图像分别输入所述n个编码器，输出对应于n种不同波长的n个特征图。
在一个优选例中，所述n个编码器为n个深度神经网络；所述分类器为深度神经网络；所述波长选择模块为基于自注意力的深度神经网络。
本申请还公开了一种内窥镜多光谱图像处理方法包括：
获取n种不同波长下获得的n个内窥镜图像；
将n种不同波长下获得的n个内窥镜图像分别输入如前文描述的内窥镜多光谱图像处理系统的编码器，获得所述系统的波长选择模块输出的对应于所述n种不同波长的n个权重；
从所述n个权重中选择大于预设阈值的权重；
输出所述大于所述预设阈值的权重对应的波长对应的内窥镜图像。
本申请还公开了一种内窥镜多光谱图像处理系统的训练方法包括：
准备多组训练数据，每组训练数据包括n种不同波长下获得的n个内窥镜图像和基于该n个内窥镜图像预先标定的分类结果；
将如前文描述的内窥镜多光谱图像处理系统作为一个整体使用所述多组数据进行训练，其中，对于每组训练数据，将所述n个内窥镜图像作为所述系统的编码器的输入，将所述分类结果作为所述系统的分类器输出。
本申请还公开了一种内窥镜多光谱图像处理方法包括：
将n种不同波长下获得的n个内窥镜图像分别输入如前文描述的内窥镜多光谱图像处理系统的n个编码器；
获取并在显示器上显示所述内窥镜多光谱图像处理系统的分类器的输出结果。
本申请实施方式中，让系统中的神经网络自动对各波长光的重要性进行排序，再反馈给系统，系统自动选择合适的波长和光强再次成像，将新的图像再次输入下一步中的分类网络进行病理分类，以及调整并输出新的图像供医生观察，从而使得医生容易看清病灶。具体地说，
1)基于自注意力模型，让神经网络自动对各波长光的重要性进行排序。
2)让系统自动学习不同光谱图片的重要性，包括：通过一个Softmax或Sigmoid激活函数归一化到0～1区间，生成一个包含n个元素的权重矩阵W。权重矩阵中权重系数w1～wn即代表了波长L1～Ln下的图片的重要性。权重越大，代表该波长对系统预测贡献越大；权重越小，代表该波长对系统预测贡献越小。
3)拼接之前，找到n个权重中小于预设阈值的权重对应的波长并将对应的波长对应的特征图置为0。因为所有的n个波长中可能会包含噪声 波长，尽管通过上述的自注意力波长选择模块可以一定程度地滤掉噪声波长，但不如将噪声删掉更为直接。所以本申请的系统将筛掉的波长对应的图像全部置为黑色。而本申请的系统的神经网络的输入是经过上述网络筛选后的波长对应的图像，即大于阈值的波长对应的图像保留，小于阈值的波长对应的图像置为黑色，输出分类结果。同样，在训练时，进行有监督训练，每组训练数据包括m种大于阈值的波长下的m个图像和小于阈值的n-m个黑色图像，以及预先标定的分类结果。
4)图像选择模块根据n个波长对应的权重调节光源。将权重w i小于阈值t的波长的光强置为0，权重wi大于阈值t的波长的光强比与权重比相同。从而剔除噪声波长，使得图像更清楚，且提高拍摄效率。
本申请的说明书中记载了大量的技术特征，分布在各个技术方案中，如果要罗列出本申请所有可能的技术特征的组合(即技术方案)的话，会使得说明书过于冗长。为了避免这个问题，本申请上述发明内容中公开的各个技术特征、在下文各个实施方式和例子中公开的各技术特征、以及附图中公开的各个技术特征，都可以自由地互相组合，从而构成各种新的技术方案(这些技术方案均应该视为在本说明书中已经记载)，除非这种技术特征的组合在技术上是不可行的。例如，在一个例子中公开了特征A+B+C，在另一个例子中公开了特征A+B+D+E，而特征C和D是起到相同作用的等同技术手段，技术上只要择一使用即可，不可能同时采用，特征E技术上可以与特征C相组合，则，A+B+C+D的方案因技术不可行而应当不被视为已经记载，而A+B+C+E的方案应当视为已经被记载。
附图说明
图1是根据本申请的内窥镜多光谱图像处理系统的结构示意图；
图2是根据本申请的一种实施方式的内窥镜多光谱图像处理系统的 结构示意图；
图3是根据本申请的内窥镜多光谱图像处理系统的一种实施例的根据特征矩阵调整光源波长得到新的图像并分类的结构示意图；
图4是根据本申请的内窥镜多光谱图像处理系统的基于CNN+XGBoost的波长选择模块的结构示意图；
图5是根据本申请的内窥镜多光谱图像处理系统的基于CNN+XGBoost的波长选择模块的一种实施例的分类器的结构示意图；
图6是根据本申请的内窥镜多光谱图像处理系统的基于OOB的波长选择模块的结构示意图；
图7是根据本申请的一种实施方式的内窥镜多光谱图像处理方法的流程示意图；
图8是根据本申请的内窥镜多光谱图像处理系统的使用流程示意图；
图9是根据本申请的内窥镜多光谱图像处理系统的息肉实时目标检测模型示意图。
具体实施方式
在以下的叙述中，为了使读者更好地理解本申请而提出了许多技术细节。但是，本领域的普通技术人员可以理解，即使没有这些技术细节和基于以下各实施方式的种种变化和修改，也可以实现本申请所要求保护的技术方案。
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请的实施方式作进一步地详细描述。
本申请的第一实施方式涉及一种内窥镜多光谱图像处理系统，如图1 所示，该内窥镜多光谱图像处理系统包括：编码器、波长选择模块、拼接模块以及分类器。
编码器，被配置为对输入的n种波长下获得的n个内窥镜图像分别进行编码，输出对应于n种不同波长的n个特征图，其中n为大于1的整数。编码器有n个，每个编码器被配置为对输入的一种波长下获得的内窥镜图像进行编码，得到对应该波长的特征图，共有n种不同波长下获得的内窥镜图像分别输入n个编码器，输出对应于n种不同波长的n个特征图。
可选的，在一个实施例中，使用本申请的系统采集同一区域不同波长窄带光照射下的图像，在二维图像的基础上增加了光谱波长的维度，成为数据立方体(Data Cube)或超立方体(Hypercubes)。所述的多光谱包括波长400nm～700nm之间的可见光，假设每隔10nm取一个中心波长进行成像，则可以获得同一区域的31张不同波长光下的图像，进而可以获得光谱维度为31的多光谱图像。在这个实施例中，n＝31，在其他的实施例中，n也可以是其他的数值，例如5、15等。
数据立方体中的每个波长下的图片L1～Ln分别输入各自的编码器1～n进行特征提取，这些编码器由深度神经网络构成，包括卷积层、下采样层、池化层、批归一化层等。这些编码器可以共享全部或部分参数，经由编码器后得到各自对应的特征图1～n。
波长选择模块，被配置为根据输入的n个特征图，输出分别权重矩阵，权重矩阵代表不同波长的特征图的重要性，权重越大，代表该波长对网络预测贡献越大；权重越小，代表该波长对系统预测贡献越小。权重矩阵用于对输入拼接模块的特征图进行控制。
波长选择模块可以用多种实现方式。
可选的，在一个实施例中，如图2所示，使用基于自注意力机制的神 经网络实现波长选择模块。多光谱图片输入波长选择模块后，波长选择模块将这些特征图在通道维度进行拼接，拼接后的特征图经过数个全连接层和激活层，最后通过一个Softmax或Sigmoid激活函数归一化到0～1区间，生成一个包含n个元素的权重矩阵W。权重矩阵中权重系数w1～wn即代表了波长L1～Ln下的图片的重要性，w1～wn均在0～1之间。
可选地，在另一个实施例中，波长选择模块基于CNN+XGBoost获得权重矩阵，如图4所示。
1)首先使用一个基于深度神经网络的模型进行特征向量的提取，该网络包括编码器、全局池化层(Global Average Pooling，GAP)、分类器。使用有监督方式训练，每组训练数据包括n种不同波长下获得的n个内窥镜图像和基于该n个内窥镜图像预先标定的分类结果。每个波长下的图片经由编码器1～n得到一个w*h*c的特征图1～n，w,h为特征图的宽和高，c为特征图通道数。特征图经通道维度拼接后，再通过全局池化层，每个特征图得到一个1*c的特征向量。即每个波长下的图像对应一个1*c的特征向量，特征向量总长度为n*c，n为波长数。
2)如图5所示，建立XGBoost模型，将上述训练好的深度神经网络模型得到的特征向量作为XGBoost模型的输入。使用有监督方式训练，每组训练数据包括n种不同波长下获得的n个内窥镜图像经过上述深度神经网络模型得到的特征向量和基于该n个内窥镜图像预先标定的分类结果。
3)利用XGBoost模型计算输入特征的重要性得分，对于输入的n*c的特征向量，得到重要性得分矩阵S＝[s 1,…,s c,s c+1,…,s 2*c,…,s (n- 1)*c+1,…,s n*c]，对于波长L i下图像对应的重要性得分[s (i-1)*c+1,…,s i*c]对其求和得到波长Li下图片对应的特征权重wi。依次计算波长L1～Ln下图像对应的特征权重w1～wn，再通过sigmoid/softmax函数将w1～wn归一化 至[0,1],得到权重矩阵W＝[w1,…,wn]。
可选地，在另一个实施例中，波长选择模块基于OOB对波长进行排序，该排序的训练过程及步骤进一步包括：
训练神经网络模型，然后使用OOB(Out of Bag)数据计算特征重要性，训练好模型之后，用Out of Bag(测试)数据进行特征重要性的量化计算。具体来说，先用训练好的模型对OOB数据进行打分，计算出准确率或者AUC(Area Under Curve)或者其他定义的评估指标；接着对OOB数据中的每个波长下的图像执行如下操作：
1)如图6所示，训练一个分类器模型，该分类器可以是基于深度神经网络的，也可以是基于传统机器学习方法(例如SVM，XGBoost等)。使用有监督的训练，每组训练数据包括n种不同波长下获得的n个内窥镜图和基于该n个内窥镜图像预先标定的分类结果；
2)用训练好的模型对测试数据进行打分，计算出准确率或者AUC(Area Under Curve)或者其他定义的评估指标；
3)将测试集中某个波长下的图像全部设为随机噪声，其余波长的图像不变，例如使得该噪声图片满足高斯分布；重新对当前测试数据集进行打分，计算评估指标；
4)计算指标变化率△，即原始测试集数据的评估指标与将某个波长下的图像设为随机噪声后的评价指标的差值。差值越大，则代表去除该波长后模型性能下降越多，该波长对预测结果影响越大；差值越小，则代表去除该波长后模型性能略微下降，该波长对预测结果影响越小。
5)依次将测试集中波长L1～Ln下的图像设为随机噪声。得到n个变化率△ 1～△ n，再通过sigmoid/softmax函数将△ 1～△ n归一化至[0,1]，得到权重矩阵W＝[w1,…,wn]。
使用权重矩阵对输入拼接模块的特征图进行控制可以有多种实现方式。
可选的，在一个实施例中，如图3所示，还包括特征图处理模块，被配置为根据权重矩阵对编码器输出的n个特征图进行筛选和/或加权处理后输出给拼接模块。
权重矩阵包括n个权重，分别对应n种不同波长的特征图。
筛选包括筛除权重矩阵的n个权重中小于预设阈值的权重对应的波长对应的特征图。设置一个预设阈值t，如果wi<t，那么认为这个wi对应波长的光谱图片Li对网络预测无显著性作用。则可以通过这个权重矩阵W筛除掉对网络预测无显著性作用的波长(wi<t)。
加权处理包括将特征图分别乘以波长选择模块输出的对应权重。将数据立方体中的每个波长下的图片L1～Ln分别输入各自的编码器1～n进行特征提取，再将特征图1～n分别乘以权重系数w1～wn，在通道维度进行拼接实现特征融合，进而送入分类器分类得到最后的病理分类结果。在一个实施例中，引入上述筛选处理，由于筛选后的波长数量从n变为了m，部分波长L i没有相应的图片，则将筛除掉的波长对应的输入像素全部置为(0,0,0)，即黑色。
可选的，这些编码器由深度神经网络构成，包括卷积层、下采样层、池化层、批归一化层等。这些编码器可以共享全部或部分参数，经由编码器后得到各自对应的特征图1～n。
可选的，在另一个实施例中，还包括预处理模块，被配置为根据权重矩阵对输入到编码器的n种波长下获得的n个内窥镜图像进行筛选和\或加权处理。
权重矩阵包括n个权重，分别对应n种不同波长的特征图。
筛选包括筛除n个权重中小于预设阈值的权重对应的波长的图像，或者将接收到的内窥镜图像中，权重小于阈值的波长对应的光强置为零。
加权处理包括根据n个权重控制每个波长的光的强度。在接收到n个内窥镜图像后，根据权重调整图片中不同波长的光的光强。可选的，光强I1：I2：…：In＝w1：w2：…：wn。
可选的，在另一个实施例中，还包括成像装置，用于拍摄内窥镜图像；成像装置被配置为根据权重矩阵对拍摄内窥镜图像所使用或接收的成像波长进行控制，包括筛选和/或光强处理以获得新的多光谱图片，再重新输入到编码器中，并进行加权处理，再输入拼接模块。
权重矩阵包括n个权重，分别对应n种不同波长的特征图。
筛选包括筛除n个权重中小于预设阈值的权重对应的波长的光。可选的，光源不使用此类权重小的波长的光进行拍摄，或者图像接收装置不接受此类权重小的波长的光。
光强处理包括根据n个权重控制每个波长的光的强度。可选的，光源通过wi的比值确定发出的各波长光的强度，光强I1：I2：…：In＝w1：w2：…：wn；或者图像接收装置通过wi的比值确定要接收的各波长光的强度，光强I1：I2：…：In＝w1：w2：…：wn。
拼接模块，被配置为对编码器输出的特征图进行拼接，得到拼接特征图。将这些特征图在通道维度进行拼接，将拼接后的特征图送入分类器得到最后的病理分类结果。
分类器，被配置为对拼接特征图进行分类，输出分类结果。可选的，在一个实施例中，分类器可以由深度神经网络构成，包括卷积层、下采样层、池化层、批归一化层、全连接层、Softmax/Sigmoid激活层等。
可选的，在一个实施例中，n个编码器为n个深度神经网络；分类器 为深度神经网络；波长选择模块为基于自注意力的深度神经网络。
系统使用多组训练数据进行有监督方式训练，每组训练数据包括n种不同波长下获得的n个内窥镜图像和基于该n个内窥镜图像预先标定的分类结果。
本申请的第二实施方式涉及一种内窥镜多光谱图像处理方法，如图7所示，包括：
在步骤701中，获取n种不同波长下获得的n个内窥镜图像。
可以有多种方式获取n个内窥镜图像。
可选的，在一个实施例中，步骤701进一步包括切换n种照射光，分别在每种照射光下获取n个内窥镜图像，每种照射光的中心波长各不相同，每种照射光的波长范围小于预定门限。
可选的，在另一个实施例中，步骤701进一步包括n次切换用于拍摄内窥镜图像的摄像机前的滤光装置以获取n个内窥镜图像，每次切换滤光装置后只有一种预定波长范围的光能够进入摄像机，每次切换滤光装置后能够进入摄像机的波长范围各不相同。
此后进入步骤702，将n种不同波长下获得的n个内窥镜图像分别输入如第一实施例的内窥镜多光谱图像处理系统的编码器，获得系统的波长选择模块输出的对应于n种不同波长的n个权重。
然后进入步骤703，从n个权重中选择大于预设阈值的权重。
然后进入步骤704，输出大于预设阈值的权重对应的波长对应的内窥镜图像。
输入到编码器的图像可以来自视频，该视频中的图像对应的波长在n种不同波长下依次切换，每连续的n个内窥镜图像中会有一个被选中输 出，不断执行上述各步骤，从而使得输出的内窥镜图像也构成连续的视频。输出的图像或视频可以显示在显示器上。
需要注意的是，本实施方式是与第一实施方式相配合的图像处理方法实施方式，第一实施方式中的技术细节可以应用于本实施方式，本实施方式中的技术细节也可以应用于第一实施方式。
本申请的第三实施方式涉及一种内窥镜多光谱图像处理系统的训练方法，包括：
准备多组训练数据，每组训练数据包括n种不同波长下获得的n个内窥镜图像和基于该n个内窥镜图像预先标定的分类结果。
将如第一实施方式的的内窥镜多光谱图像处理系统作为一个整体使用多组数据进行训练，其中，对于每组训练数据，将n个内窥镜图像作为系统的编码器的输入，将分类结果作为系统的分类器输出。
需要注意的是，本实施方式是与第一实施方式相配合的训练方法实施方式，第一实施方式中的技术细节可以应用于本实施方式，本实施方式中的技术细节也可以应用于第一实施方式。
本申请的第四实施方式涉及一种内窥镜多光谱图像处理方法，包括：
将n种不同波长下获得的n个内窥镜图像分别输入如第一实施方式的内窥镜多光谱图像处理系统的n个编码器。
获取并在显示器上内窥镜多光谱图像处理系统的分类器的输出结果。
需要注意的是，本实施方式是与第一实施方式相配合的图像处理方法实施方式，第一实施方式中的技术细节可以应用于本实施方式，本实施方式中的技术细节也可以应用于第一实施方式。
为了能够更好地理解本申请的技术方案，下面结合一个具体的例子来进行说明，该例子中罗列的细节主要是为了便于理解，不作为对本申请保护范围的限制。
如图8所示，其中，CAD系统即为本申请第一实施方式所描述的系统，本系统主要包括以下步骤：
1)医生使用在白光模式进行息肉筛查，此系统实时检测息肉所在位置。图9是一个息肉实时目标检测模型。白光下的图像x作为输入，输入一个编码器进行特征提取，特征提取后得到的特征图再经由一个检测器回归得到息肉所在位置的坐标y。其中编码器和检测器由深度神经网络构成，包括卷积层、下采样层、上采样层、池化层、批归一化层、激活层等。
2)发现息肉后，系统会在监视器上用红色框提示医生息肉所在的区域，并发出警报声，图像处理中心反馈给光源，切换到多光谱模式，获得同一区域连续的多个波长窄带光下的多光谱图像。
3)本系统实时分析多光谱图像，通过建立好的光谱特征排序与可解释性模型给出病理性质判断。同时根据此模型自动选择最为合适的单个或多个波长的窄带光、以及各个波长光的强度，通过图像处理中心反馈至光源进行切换。
4)此系统通过新的多光谱图像再次判断病理性质，结合医生在此新图像下的判断给出最后的病理性质判断。
需要说明的是，在本专利的申请文件中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的 包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。本专利的申请文件中，如果提到根据某要素执行某行为，则是指至少根据该要素执行该行为的意思，其中包括了两种情况：仅根据该要素执行该行为、和根据该要素和其它要素执行该行为。多个、多次、多种等表达包括2个、2次、2种以及2个以上、2次以上、2种以上。
本说明书包括本文所描述的各种实施例的组合。对“一个实施例”或特定实施例等的单独提及不一定是指相同的实施例；然而，除非指示为是互斥的或者本领域技术人员很清楚是互斥的，否则这些实施例并不互斥。应当注意的是，除非上下文另外明确指示或者要求，否则在本说明书中以非排他性的意义使用“或者”一词。
在本申请提及的所有文献都被认为是整体性地包括在本申请的公开内容中，以便在必要时可以作为修改的依据。此外应理解，在阅读了本申请的上述公开内容之后，本领域技术人员可以对本申请作各种改动或修改，这些等价形式同样落于本申请所要求保护的范围。

Claims (12)

  1. 一种内窥镜多光谱图像处理系统，其特征在于，包括：
    编码器，被配置为对输入的n种波长下获得的n个内窥镜图像分别进行编码，输出对应于n种不同波长的n个特征图，其中n为大于1的整数；
    波长选择模块，被配置为根据输入的所述n个特征图，输出权重矩阵，所述权重矩阵代表不同波长的特征图的重要性，所述权重矩阵用于对输入拼接模块的特征图进行控制；
    所述拼接模块，被配置为对所述编码器输出的特征图进行拼接，得到拼接特征图；
    分类器，被配置为对所述拼接特征图进行分类，输出分类结果。
  2. 如权利要求1所述的内窥镜多光谱图像处理系统，其特征在于，还包括特征图处理模块，被配置为根据所述权重矩阵对所述编码器输出的n个特征图进行筛选和/或加权处理后输出给所述拼接模块。
  3. 如权利要求2所述的内窥镜多光谱图像处理系统，其特征在于，
    所述权重矩阵包括n个权重，分别对应n种不同波长的特征图；
    所述筛选包括筛除所述权重矩阵的n个权重中小于预设阈值的权重对应的波长对应的所述特征图；
    所述加权处理包括将所述特征图分别乘以所述波长选择模块输出的对应权重。
  4. 如权利要求1所述的内窥镜多光谱图像处理系统，其特征在于，还包括预处理模块，被配置为根据所述权重矩阵对输入到所述编码器的n种波长下获得的n个内窥镜图像进行筛选和\或加权处理。
  5. 如权利要求1所述的内窥镜多光谱图像处理系统，其特征在于，还包括成像装置，用于拍摄内窥镜图像；
    所述成像装置被配置为根据所述权重矩阵对拍摄内窥镜图像所使用或接收的成像波长进行控制。
  6. 如权利要求5所述的内窥镜多光谱图像处理系统，其特征在于，
    所述控制包括筛选和/或光强处理；
    所述权重矩阵包括n个权重，分别对应n种不同波长的特征图；
    所述筛选包括筛除所述n个权重中小于预设阈值的权重对应的波长的光；
    所述光强处理包括根据所述n个权重控制每个波长的光的强度。
  7. 如权利要求1所述的内窥镜多光谱图像处理系统，其特征在于，所述系统使用多组训练数据进行有监督方式训练，每组训练数据包括n种不同波长下获得的n个内窥镜图像和基于该n个内窥镜图像预先标定的分类结果。
  8. 如权利要求1所述的内窥镜多光谱图像处理系统，其特征在于，
    所述编码器有n个，每个编码器被配置为对输入的一种波长下获得的内窥镜图像进行编码，得到对应该波长的特征图，共有n种不同波长下获得的内窥镜图像分别输入所述n个编码器，输出对应于n种不同波长的n个特征图。
  9. 如权利要求8所述的内窥镜多光谱图像处理系统，其特征在于，所述n个编码器为n个深度神经网络；所述分类器为深度神经网络；所述波长选择模块为基于自注意力的深度神经网络。
  10. 一种内窥镜多光谱图像处理方法，其特征在于，包括：
    获取n种不同波长下获得的n个内窥镜图像；
    将n种不同波长下获得的n个内窥镜图像分别输入如权利要求1所述的内窥镜多光谱图像处理系统的编码器，获得所述系统的波长选择模块输出的对应于所述n种不同波长的n个权重；
    从所述n个权重中选择大于预设阈值的权重；
    输出所述大于所述预设阈值的权重对应的波长对应的内窥镜图像。
  11. 一种内窥镜多光谱图像处理系统的训练方法，其特征在于，包括：
    准备多组训练数据，每组训练数据包括n种不同波长下获得的n个内窥镜图像和基于该n个内窥镜图像预先标定的分类结果；
    将如权利要求1所述的内窥镜多光谱图像处理系统作为一个整体使用所述多组数据进行训练，其中，对于每组训练数据，将所述n个内窥镜图像作为所述系统的编码器的输入，将所述分类结果作为所述系统的分类器输出。
  12. 一种内窥镜多光谱图像处理方法，其特征在于，包括：
    将n种不同波长下获得的n个内窥镜图像分别输入如权利要求1所述的内窥镜多光谱图像处理系统的n个编码器；
    获取并在显示器上显示所述内窥镜多光谱图像处理系统的分类器的输出结果。
PCT/CN2022/089022 2021-04-27 2022-04-25 内窥镜多光谱图像处理系统及处理和训练方法 WO2022228396A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110461653.X 2021-04-27
CN202110461653.XA CN115245312A (zh) 2021-04-27 2021-04-27 内窥镜多光谱图像处理系统及处理和训练方法

Publications (1)

Publication Number Publication Date
WO2022228396A1 true WO2022228396A1 (zh) 2022-11-03

Family

ID=83697430

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/089022 WO2022228396A1 (zh) 2021-04-27 2022-04-25 内窥镜多光谱图像处理系统及处理和训练方法

Country Status (2)

Country Link
CN (1) CN115245312A (zh)
WO (1) WO2022228396A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117204796B (zh) * 2023-11-09 2024-02-13 哈尔滨海鸿基业科技发展有限公司 一种腹腔内窥镜多光谱成像方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160307073A1 (en) * 2015-04-20 2016-10-20 Los Alamos National Security, Llc Change detection and change monitoring of natural and man-made features in multispectral and hyperspectral satellite imagery
CN107169535A (zh) * 2017-07-06 2017-09-15 谈宜勇 生物多光谱图像的深度学习分类方法及装置
CN111695407A (zh) * 2020-04-23 2020-09-22 西安电子科技大学 基于多光谱融合的性别识别方法、系统、存储介质及终端
CN111898427A (zh) * 2020-06-22 2020-11-06 西北工业大学 一种基于特征融合深度神经网络的多光谱行人检测方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160307073A1 (en) * 2015-04-20 2016-10-20 Los Alamos National Security, Llc Change detection and change monitoring of natural and man-made features in multispectral and hyperspectral satellite imagery
CN107169535A (zh) * 2017-07-06 2017-09-15 谈宜勇 生物多光谱图像的深度学习分类方法及装置
CN111695407A (zh) * 2020-04-23 2020-09-22 西安电子科技大学 基于多光谱融合的性别识别方法、系统、存储介质及终端
CN111898427A (zh) * 2020-06-22 2020-11-06 西北工业大学 一种基于特征融合深度神经网络的多光谱行人检测方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JIN YAN-LIANG;GE FEI-YANG: "Improved Fusion Method Based on Ambient Illumination Condition for Multispectral Pedestrian Detection", MICROELECTRONICS & COMPUTER, vol. 38, no. 1, 5 January 2021 (2021-01-05), pages 27 - 32, XP055981521, ISSN: 1000-7180, DOI: 10.19304/j.cnki.issn1000-7180.2021.01.006 *
LIU QI; LI ZHENGTAO; SHUAI SHUAI; SUN QIZHEN: "Spectral group attention networks for hyperspectral image classification with spectral separability analysis", INFRARED PHYSICS AND TECHNOLOGY, vol. 108, 5 May 2020 (2020-05-05), GB , pages 1 - 10, XP086169712, ISSN: 1350-4495, DOI: 10.1016/j.infrared.2020.103340 *

Also Published As

Publication number Publication date
CN115245312A (zh) 2022-10-28

Similar Documents

Publication Publication Date Title
JP6657480B2 (ja) 画像診断支援装置、画像診断支援装置の作動方法および画像診断支援プログラム
JP2021532881A (ja) マルチスペクトル情報を用いた拡張画像化のための方法およびシステム
JP7005767B2 (ja) 内視鏡画像認識装置、内視鏡画像学習装置、内視鏡画像学習方法及びプログラム
EP1576920B1 (en) Imaging device
JP7062068B2 (ja) 画像処理方法及び画像処理装置
JP2021532891A (ja) マルチスペクトル情報を用いた観血的治療における拡張画像化のための方法およびシステム
JP7252970B2 (ja) 医用画像処理装置、内視鏡システム、及び医用画像処理装置の作動方法
WO2020162275A1 (ja) 医療画像処理装置、内視鏡システム、及び医療画像処理方法
JP7308258B2 (ja) 医療画像処理装置及び医療画像処理装置の作動方法
US20210076917A1 (en) Image processing apparatus, endoscope system, and image processing method
JP7289296B2 (ja) 画像処理装置、内視鏡システム及び画像処理装置の作動方法
US11978550B2 (en) Endoscopic image learning device, endoscopic image learning method, endoscopic image learning program, and endoscopic image recognition device
CN114372951A (zh) 基于图像分割卷积神经网络的鼻咽癌定位分割方法和系统
WO2022228396A1 (zh) 内窥镜多光谱图像处理系统及处理和训练方法
JP2023026480A (ja) 医療画像処理装置、内視鏡システム、及び医療画像処理装置の作動方法
US20220285010A1 (en) Medical image processing apparatus, medical image processing method, and program
JP2022183277A (ja) 内視鏡装置、内視鏡プロセッサ、及び内視鏡装置の作動方法
Liedlgruber et al. A summary of research targeted at computer-aided decision support in endoscopy of the gastrointestinal tract
CN115279247A (zh) 处理系统、图像处理方法以及学习方法
JP7346596B2 (ja) 画像処理システム及びプログラム
CN114376491A (zh) 内窥镜成像装置及其方法、系统以及电子设备
CN214231268U (zh) 内窥镜成像装置和电子设备
WO2024009631A1 (ja) 画像処理装置及び画像処理装置の作動方法
US20230389774A1 (en) Medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program
JP7449004B2 (ja) 周波数帯域を用いたハイパースペクトルによる物体画像の検出方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22794866

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22794866

Country of ref document: EP

Kind code of ref document: A1