WO2023221149A1 - Cnv focus forging method, apparatus and system based on retinal oct image - Google Patents

Cnv focus forging method, apparatus and system based on retinal oct image Download PDF

Info

Publication number
WO2023221149A1
WO2023221149A1 PCT/CN2022/094354 CN2022094354W WO2023221149A1 WO 2023221149 A1 WO2023221149 A1 WO 2023221149A1 CN 2022094354 W CN2022094354 W CN 2022094354W WO 2023221149 A1 WO2023221149 A1 WO 2023221149A1
Authority
WO
WIPO (PCT)
Prior art keywords
cnv
lesions
retinal
oct images
features
Prior art date
Application number
PCT/CN2022/094354
Other languages
French (fr)
Chinese (zh)
Inventor
陈新建
王景涛
蔡茂江
Original Assignee
苏州比格威医疗科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏州比格威医疗科技有限公司 filed Critical 苏州比格威医疗科技有限公司
Publication of WO2023221149A1 publication Critical patent/WO2023221149A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/12—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes
    • A61B3/1225—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes using coherent radiation

Definitions

  • the generated labels are classification labels, segmentation labels or detection labels, and the labels are converted based on hierarchical labels;
  • the retinal OCT image processing model is a classification model, segmentation model or detection model.
  • the present invention provides a device for forging CNV lesions based on retinal OCT images, including:
  • the CNV discriminator is trained using meta-learning on real retinal OCT images with CNV lesions
  • this invention can forge lesions in a large number of retinal OCT images without CNV lesions, and can infinitely expand the retinal OCT images and labels with CNV lesions, effectively solving the problems of small amount of data and difficulty in labeling. .
  • Step (5) Based on the location, size and contour of the forged CNV lesions calculated in steps (3) and (4), perform sampling and pixel transformation operations on the retinal OCT image and its layered gold standard respectively. , obtain retinal OCT images and hierarchical labels with CNV lesions.
  • the generated labels are classification labels
  • the retinal OCT image processing model is a classification model.
  • the generated labels may also be segmentation labels or detection labels
  • the retinal OCT image processing model may also be a segmentation model or a detection model.
  • the classification tags, segmentation tags or detection tags can be converted by using the aforementioned hierarchical features through existing technologies.
  • Step 2 Forgery of CNV lesions and screening of generated samples
  • step 1 Use the CNV discriminator trained in step 1 to determine whether the generated OCT images and label data with CNV are qualified. If it is unqualified, return to step (1), destroy and regenerate; if it is qualified, go to step three;
  • the model training module is used to send qualified data output by the CNV discriminator and real retinal OCT images with CNV lesions into the deep learning model that processes OCT images for training, and obtain a retinal OCT image processing model.
  • the model training module receives qualified data output by the CNV discriminator and real retinal OCT images with CNV lesions, and performs training to obtain a retinal OCT image processing model.
  • the method for extracting true CNV features includes the following steps:
  • sampling and pixel transformation operations of the layered gold standard are consistent with those of OCT images.

Abstract

Provided are a CNV focus forging method, apparatus, and system based on a retinal OCT image. The method comprises: using a real retinal OCT image with a CNV focus and a corresponding label to train a CNV discriminator; based on the real retinal OCT image with the CNV focus, extracting CNV real characteristics; acquiring CNV simulation characteristics; based on the CNV real characteristics and the CNV simulation characteristics, forging CNV focuses on retinal OCT images without CNV focuses to generate in batches retinal OCT images with CNV focus and labels, and sending the retinal OCT images with the CNV focuses and the labels into the CNV discriminator; and sending qualified data output by the CNV discriminator and the real retinal OCT image with the CNV focus into a deep learning model for training to obtain a retinal OCT image processing model. According to the method, apparatus, and system, the characteristics of the real CNV focus can be extracted from a small number of real retinal OCT images with the CNV focuses, and by combining the simulation characteristics of the CNV focuses, the CNV focuses are forged in a large number of retinal OCT images without CNV focuses, so that the retinal OCT image processing model with high accuracy is generated.

Description

基于视网膜OCT图像的CNV病灶伪造方法、装置及系统CNV lesion forgery method, device and system based on retinal OCT images 技术领域Technical field
本发明属于视网膜图像处理技术领域，具体涉及一种基于视网膜OCT图像的CNV病灶伪造方法、装置及系统。The invention belongs to the technical field of retinal image processing, and specifically relates to a method, device and system for forging CNV lesions based on retinal OCT images.
背景技术Background technique
光学相干断层扫描(Optical Coherence Tomography，OCT)是一种形成眼底影像的成像技术，由于其能反应眼底不同生理结构对入射的弱相干光的反射、散射特性，形成的三维影像具有深度层面的信息，相比与眼底彩照等影像，其具有独特的优势。Optical Coherence Tomography (OCT) is an imaging technology that forms fundus images. Because it can reflect the reflection and scattering characteristics of different physiological structures of the fundus on incident weak coherent light, the three-dimensional image formed has depth-level information. , compared with images such as fundus color photos, it has unique advantages.
脉络膜新生血管(Choroidal Neo Vascularization，CNV)，又称视网膜下新生血管，是年龄相关性黄斑变性、中央渗出性脉络膜视网膜病变、特发性脉络膜新生血管、病理性近视黄斑变性、眼组织胞浆病综合征等多种眼内疾病的一种基本病理改变。它常累及黄斑，对中央视力造成严重损害。Choroidal Neovascularization (CNV), also known as subretinal neovascularization, is a disease of age-related macular degeneration, central exudative chorioretinopathy, idiopathic choroidal neovascularization, pathological myopic macular degeneration, and eye tissue cytoplasm. A basic pathological change in various intraocular diseases such as disease syndrome. It often involves the macula and causes severe damage to central vision.
针对视网膜OCT图像CNV处理的问题，已经有大量文献报道了各种不同的算法，主要包括基于深度学习的有监督和无监督方法，基于图论的图割/图搜索方法等。基于深度学习视网膜OCT图像CNV处理方法省略了人工设计特征的繁琐过程，采用卷积和池化等操作从原始图像数据中学习到抽象、深层次的特征，但是，深度学习算法的模型训练通常需要大量有标注的训练数据进行模型参数优化，与自然图像相比，带标注的且包含CNV病灶的视网膜OCT影像非常稀缺，并且需要有丰富的临床经验的专家医生手动完成数据标注，标注成本极高，同时难以保证训练数据与实际测试应用数据之间的独立同分布性，使得模型易产生过拟合和泛化性不足的问题。另外，不同机器拍摄的OCT图像，由于波长、算法等原因，其病灶的特征也不同，因此，通常情况下只能针对一种机型拍摄的OCT图像训练模型，如果跨机型训练模型，会降低模型的处理效果；另外，用一种机型拍摄的OCT图形训练的模型，通常也无法应用到另外一种机型拍摄的OCT图像。然而，特定机型且有标注的带CNV病灶的视网膜OCT图像，多少情况下是非常匮乏的。Regarding the problem of CNV processing of retinal OCT images, a large number of literatures have reported various different algorithms, mainly including supervised and unsupervised methods based on deep learning, graph cut/graph search methods based on graph theory, etc. The retinal OCT image CNV processing method based on deep learning omits the tedious process of manually designing features, and uses operations such as convolution and pooling to learn abstract and deep features from the original image data. However, model training of deep learning algorithms usually requires A large amount of annotated training data is used to optimize model parameters. Compared with natural images, annotated retinal OCT images containing CNV lesions are very scarce, and expert doctors with rich clinical experience are required to manually complete data annotation, and the annotation cost is extremely high. , and at the same time, it is difficult to ensure the independent and identical distribution between the training data and the actual test application data, making the model prone to overfitting and insufficient generalization. In addition, OCT images taken by different machines have different characteristics of lesions due to reasons such as wavelength and algorithm. Therefore, usually the model can only be trained on OCT images taken by one model. If the model is trained across models, it will cause problems. Reduce the processing effect of the model; in addition, a model trained with OCT images captured by one model of camera usually cannot be applied to OCT images captured by another model of camera. However, labeled retinal OCT images with CNV lesions of specific models are in many cases very scarce.
面对带CNV病灶的视网膜OCT图像匮乏的问题，现有的解决方式包括：收集和标注更多的数据，利用预训练模型以及对抗网络生成模型等等。收集和标注更多的数据需要花费大量的时间和金钱，而且和其他医疗图像一样，视网膜OCT图像同样难以收集，尤其是特定机型且带有疾病的OCT图像。由于预训练的模型结构灵活性差、应用场景受限，难以针对视网膜OCT图像处理的具体问题，找到合适预训练模型进行微调。利用对抗网络生成模型生成的虚拟数据，难以捕捉到CNV复杂的病灶特征，且生成的特征不具有可 解释性，难以提高视网膜OCT图像处理模型的准确率。Facing the problem of lack of retinal OCT images with CNV lesions, existing solutions include: collecting and annotating more data, using pre-trained models and adversarial network generation models, etc. It takes a lot of time and money to collect and label more data, and like other medical images, retinal OCT images are also difficult to collect, especially OCT images of specific models and diseases. Due to the poor structural flexibility of the pre-trained model and the limited application scenarios, it is difficult to find a suitable pre-trained model for fine-tuning based on the specific problems of retinal OCT image processing. It is difficult to capture the complex lesion characteristics of CNV using the virtual data generated by the adversarial network generation model, and the generated features are not interpretable, making it difficult to improve the accuracy of the retinal OCT image processing model.
发明内容Contents of the invention
针对上述问题，本发明提出一种基于视网膜OCT图像的CNV病灶伪造方法、装置及系统，能够从少量真实的带CNV病灶的视网膜OCT图像中提取真实CNV病灶的特征，并基于提取到的真实CNV病灶的特征，在大量的不带CNV病灶的视网膜OCT图像伪造CNV病灶，最终生成高准确率的视网膜OCT图像处理模型。In response to the above problems, the present invention proposes a CNV lesion forgery method, device and system based on retinal OCT images, which can extract the characteristics of real CNV lesions from a small number of real retinal OCT images with CNV lesions, and based on the extracted real CNV The characteristics of the lesions are used to forge CNV lesions in a large number of retinal OCT images without CNV lesions, and finally a high-accuracy retinal OCT image processing model is generated.
为了实现上述技术目的，达到上述技术效果，本发明通过以下技术方案实现：In order to achieve the above technical objectives and achieve the above technical effects, the present invention is implemented through the following technical solutions:
第一方面，本发明提供了一种基于视网膜OCT图像的CNV病灶伪造方法，包括：In a first aspect, the present invention provides a method for forging CNV lesions based on retinal OCT images, including:
利用真实的带CNV病灶的视网膜OCT图像，以及对应的标签，训练出CNV判别器；Use real retinal OCT images with CNV lesions and corresponding labels to train a CNV discriminator;
基于真实的带CNV病灶的视网膜OCT图像，提取出CNV真实特征；Based on real retinal OCT images with CNV lesions, the real features of CNV are extracted;
获取CNV模拟特征；Obtain CNV simulation characteristics;
基于所述CNV真实特征和CNV模拟特征，在不带CNV病灶的视网膜OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器；Based on the CNV real features and CNV simulated features, forge CNV lesions on retinal OCT images without CNV lesions, batch generate retinal OCT images and labels with CNV lesions, and send them to the CNV discriminator;
将CNV判别器输出的合格数据和真实的带CNV病灶的视网膜OCT图像送入深度学习模型进行训练，得到视网膜OCT图像处理模型。The qualified data output by the CNV discriminator and the real retinal OCT images with CNV lesions are sent to the deep learning model for training, and the retinal OCT image processing model is obtained.
可选地，所述真实的带CNV病灶的视网膜OCT图像数量小于第一阈值；所述CNV模拟特征的数量大于第二阈值，所述第一阈值小于第二阈值。Optionally, the number of real retinal OCT images with CNV lesions is less than a first threshold; the number of CNV simulation features is greater than a second threshold, and the first threshold is less than the second threshold.
可选地，所述CNV判别器的训练方法包括：Optionally, the training method of the CNV discriminator includes:
在自然数据集上使用2-ways-5query-20shot方式训练出元学习器和二分类器；Use the 2-ways-5query-20shot method to train the meta-learner and binary classifier on natural data sets;
基于所述元学习器和二分类器，在所述真实的带CNV病灶的视网膜OCT图像上继续训练，最终得到CNV判别器。Based on the meta-learner and binary classifier, training is continued on the real retinal OCT images with CNV lesions, and finally a CNV discriminator is obtained.
可选地，所述CNV真实特征的提取方法包括：Optionally, the extraction method of CNV real features includes:
从所述真实的带CNV病灶的视网膜OCT图像上提取CNV病灶的大小、轮廓和位置特征；Extract the size, contour and location features of CNV lesions from the real retinal OCT images with CNV lesions;
对CNV病灶进行仿射变换和弹性变形，并提取CNV病灶的大小、轮廓和位置特征，进行对CNV病灶的大小、轮廓和位置特征的扩充。Perform affine transformation and elastic deformation on CNV lesions, and extract the size, contour and position features of CNV lesions to expand the size, contour and position features of CNV lesions.
可选地，所述CNV模拟特征的获取方法包括：Optionally, the method for obtaining the CNV simulation features includes:
基于CNV特征模拟算法，自动生成CNV病灶的大小、轮廓和位置特征。Based on the CNV feature simulation algorithm, the size, contour and location features of CNV lesions are automatically generated.
可选地，所述带CNV病灶的视网膜OCT图像及标签的生成方法包括：Optionally, the method for generating retinal OCT images and labels with CNV lesions includes:
获取视网膜OCT图像及其视网膜分层金标准数据，基于所述视网膜分层金标准数据定位出视网膜各层的位置，并计算出视网膜各层的特征；Obtain the retinal OCT image and its retinal layering gold standard data, locate the positions of each retinal layer based on the retinal layering gold standard data, and calculate the characteristics of each retinal layer;
选择一组CNV真实特征或CNV模拟特征；Select a set of CNV real features or CNV simulated features;
若基于所述视网膜各层的特征，判断出该视网膜OCT图像适合伪造CNV病灶，则根据CNV真实特征或CNV模拟特征的位置特征，以及定位出的视网膜各层的位置，计算出适合在该张视网膜OCT图像上伪造CNV病灶的位置；If it is determined that the retinal OCT image is suitable for forging CNV lesions based on the characteristics of each retinal layer, then based on the position characteristics of the CNV real features or CNV simulated features and the located positions of each retinal layer, a calculation method is calculated that is suitable for forging CNV lesions in the image. Location of spurious CNV lesions on retinal OCT images;
根据计算出的视网膜各层厚度，以及CNV真实特征或CNV模拟特征的大小和轮廓特征，计算出适合在该张视网膜OCT图像上伪造病灶的大小和轮廓；Based on the calculated thickness of each retinal layer, as well as the size and contour characteristics of the real CNV features or CNV simulated features, calculate the size and contour suitable for forging the lesion on the retinal OCT image;
基于计算出的适合伪造CNV病灶的位置、大小和轮廓，分别对获取到的OCT图像及其分层金标准进行采样和像素变换，得到带CNV病灶的视网膜OCT图像及分层标签。Based on the calculated location, size and contour suitable for forged CNV lesions, the obtained OCT image and its layered gold standard were sampled and pixel transformed respectively to obtain the retinal OCT image with CNV lesions and layered labels.
可选地，所述对获取到的OCT图像及其分层金标准进行采样和像素变换，包括以下步骤：Optionally, the sampling and pixel transformation of the acquired OCT image and its layered gold standard include the following steps:
基于计算出的适合伪造CNV病灶的位置、大小和轮廓，确定出伪造的CNV病灶；Determine the falsified CNV lesion based on the calculated location, size, and contour of the falsified CNV lesion suitable for the falsified CNV lesion;
分别计算伪造的CNV病灶轮廓内的各点P 2(w p2,h p2)的像素值，其中，w p2为横坐标，h p2为纵坐标；所述像素值的计算包括以下子步骤： The pixel values of each point P 2 (w p2 , h p2 ) within the forged CNV lesion outline are calculated respectively, where w p2 is the abscissa and h p2 is the ordinate; the calculation of the pixel value includes the following sub-steps:
定义P 1(w p1,h p1)和P 3(w p3,h p3)分别为伪造的CNV病灶轮廓上某两点的位置，其中w p1和w p3分别为横坐标，h p1和h p3分别为纵坐标，w p1＝w p3＝w p2；Q(w q,h m)为视网膜外丛装层与内核分割线上的某一点，w q为该点的横坐标，h m为该点的纵坐标，其中w q＝w p3；采样点T(w t,h t)的像素值用Q表示，点P 2(w p2,h p2)像素值用
Figure PCTCN2022094354-appb-000001
表示，则
Figure PCTCN2022094354-appb-000002
e为0～10之间的随机整数； Define P 1 (w p1 , h p1 ) and P 3 (w p3 , h p3 ) as the positions of two points on the contour of the forged CNV lesion, where w p1 and w p3 are the abscissas, h p1 and h p3 respectively. are the ordinates respectively, w p1 = w p3 = w p2 ; Q (w q , h m ) is a point on the dividing line between the outer plexus layer and the inner core of the retina, w q is the abscissa of the point, and h m is The ordinate of the point, where w q =w p3 ; the pixel value of the sampling point T (w t , h t ) is represented by Q, and the pixel value of the point P 2 (w p2 , h p2 ) is represented by
Figure PCTCN2022094354-appb-000001
means, then
Figure PCTCN2022094354-appb-000002
e is a random integer between 0 and 10;
计算出采样点T(w t,h t)的位置，进而获得采样点T(w t,h t)的像素值，并根据采样点T(w t,h t)的像素值计算出点P 2(w p2,h p2)的像素值，其中，采样点T(w t,h t)的位置的计算公式为： Calculate the position of the sampling point T(w t ,h t ), then obtain the pixel value of the sampling point T(w t ,h t ), and calculate the point P based on the pixel value of the sampling point T(w t ,h t ) 2 (w p2 ,h p2 ) pixel value, where the calculation formula for the position of sampling point T (w t ,h t ) is:
w t＝w p2±σ wt ＝ wp2 ±σ
Figure PCTCN2022094354-appb-000003
Figure PCTCN2022094354-appb-000003
其中，σ是随机整数。where σ is a random integer.
可选地，生成的标签为分类标签、分割标签或检测标签，所述标签基于分层标签转换而成；所述视网膜OCT图像处理模型为分类模型、分割模型或检测模型。Optionally, the generated labels are classification labels, segmentation labels or detection labels, and the labels are converted based on hierarchical labels; the retinal OCT image processing model is a classification model, segmentation model or detection model.
第二方面，本发明提供了一种基于视网膜OCT图像的CNV病灶伪造装置，包括：In a second aspect, the present invention provides a device for forging CNV lesions based on retinal OCT images, including:
训练模块，用于利用所述真实的带CNV病灶的视网膜OCT图像，以及对应的标签， 训练出CNV判别器；A training module used to train a CNV discriminator using the real retinal OCT images with CNV lesions and corresponding labels;
提取模块，用于基于所述真实的带CNV病灶的视网膜OCT图像，提取出CNV真实特征；An extraction module, configured to extract real features of CNV based on the real retinal OCT image with CNV lesions;
获取模块，用于获取CNV模拟特征；Acquisition module, used to obtain CNV simulation features;
伪造模块，用于基于所述CNV真实特征和CNV模拟特征，在不带CNV病灶的视网膜OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器；A forgery module, used to forge CNV lesions on retinal OCT images without CNV lesions based on the CNV real features and CNV simulated features, batch generate retinal OCT images and labels with CNV lesions, and send them to the CNV discriminator ;
模型训练模块，用于将CNV判别器输出的合格数据和真实的带CNV病灶的视网膜OCT图像送入处理OCT图像的深度学习模型进行训练，得到视网膜OCT图像处理模型。The model training module is used to send qualified data output by the CNV discriminator and real retinal OCT images with CNV lesions into the deep learning model that processes OCT images for training, and obtain a retinal OCT image processing model.
第三方面，本发明提供了一种基于视网膜OCT图像的CNV病灶伪造装置，包括：CNV特征生成模块、CNV判别器、CNV病灶伪造模块和模型训练模块；In a third aspect, the present invention provides a CNV lesion forgery device based on retinal OCT images, including: a CNV feature generation module, a CNV discriminator, a CNV lesion forgery module and a model training module;
所述CNV特征生成模块包括CNV特征提取器、CNV特征模拟器和特征扩充器，所述CNV特征提取器从真实的带CNV病灶的视网膜OCT图像中提取CNV病灶的CNV真实特征，然后输入特征扩充器进行仿射和弹性变换，批量扩充CNV病灶的特征；所述CNV特征模拟器批量生成CNV模拟特征；The CNV feature generation module includes a CNV feature extractor, a CNV feature simulator and a feature expander. The CNV feature extractor extracts the CNV real features of CNV lesions from real retinal OCT images with CNV lesions, and then inputs the feature expansion The CNV feature simulator performs affine and elastic transformation to batch-expand the features of CNV lesions; the CNV feature simulator batch-generates CNV simulation features;
所述CNV判别器是在真实的带CNV病灶的视网膜OCT图像上使用元学习的方式训练获得；The CNV discriminator is trained using meta-learning on real retinal OCT images with CNV lesions;
所述CNV病灶伪造模块使用CNV特征生成模块输出的CNV真实特征和CNV模拟特征，在大量无CNV病灶的OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器进行判断；The CNV lesion forgery module uses the CNV real features and CNV simulated features output by the CNV feature generation module to forge CNV lesions on a large number of OCT images without CNV lesions, batch generate retinal OCT images and labels with CNV lesions, and send them to the institute. The CNV discriminator is used to judge;
所述模型训练模块接收CNV判别器输出的合格的数据和真实的带CNV病灶的视网膜OCT图像，并进行训练，得到视网膜OCT图像处理模型。The model training module receives qualified data output by the CNV discriminator and real retinal OCT images with CNV lesions, and performs training to obtain a retinal OCT image processing model.
第四方面，本发明提供了一种基于视网膜OCT图像的CNV病灶伪造系统，包括处理器及存储介质；In a fourth aspect, the present invention provides a CNV lesion forgery system based on retinal OCT images, including a processor and a storage medium;
所述存储介质用于存储指令；The storage medium is used to store instructions;
所述处理器用于根据所述指令进行操作以执行根据第一方面任一项所述方法的步骤。The processor is configured to operate according to the instructions to perform the steps of the method according to any one of the first aspects.
与现有技术相比，本发明的有益效果：Compared with the existing technology, the beneficial effects of the present invention are:
本发明能够从少量真实的带CNV病灶的视网膜OCT图像中提取真实CNV病灶的特征，并基于提取到的真实CNV病灶的特征，在大量的不带CNV病灶的视网膜OCT图像伪造CNV病灶，最终生成高准确率的视网膜OCT图像处理模型。The present invention can extract the characteristics of real CNV lesions from a small number of real retinal OCT images with CNV lesions, and based on the extracted characteristics of real CNV lesions, forge CNV lesions in a large number of retinal OCT images without CNV lesions, and finally generate Highly accurate retinal OCT image processing model.
本发明使用基于元学习MAML算法的模型训练方式训练CNV判别器，能够有效地避 免由于缺少数据导致模型过拟合的问题。The present invention uses a model training method based on the meta-learning MAML algorithm to train the CNV discriminator, which can effectively avoid the problem of model overfitting due to lack of data.
本发明通过提取CNV病灶的CNV真实特征，并进行弹性变形和仿射变换，以及算法自动模拟生成CNV模拟特征，可以最大程度的覆盖CNV特征的多样性。The present invention can cover the diversity of CNV features to the greatest extent by extracting the true CNV features of CNV lesions, performing elastic deformation and affine transformation, and automatically generating CNV simulation features through algorithm simulation.
本发明基于CNV真实特征和CNV模拟特征，在大量的不带CNV病灶的视网膜OCT图像伪造病灶，可以无限扩充带CNV病灶的视网膜OCT图像及标签，有效的解决了数据量少、标注困难的问题。Based on the real characteristics of CNV and the simulated characteristics of CNV, this invention can forge lesions in a large number of retinal OCT images without CNV lesions, and can infinitely expand the retinal OCT images and labels with CNV lesions, effectively solving the problems of small amount of data and difficulty in labeling. .
本发明通过CNV判别器判断批量生成的样本和标签是否和合格，保证了生成的样本数据和真实样本的相似性，显著提高判别的效率，并极大缩减人工判断所花费的时间和精力。This invention uses a CNV discriminator to determine whether batch-generated samples and labels are qualified, ensuring the similarity between the generated sample data and real samples, significantly improving the efficiency of discrimination, and greatly reducing the time and energy spent on manual judgment.
本发明中的标签和样本的批量生成是同时进行的，生成的标签既可以是分类标签，也可以是分割、检测的标签，因此生成的样本可以用于分类模型，也可以用于分割、检测模型，大大丰富了视网膜OCT图像处理范围。The batch generation of labels and samples in the present invention is carried out at the same time. The generated labels can be either classification labels or segmentation and detection labels. Therefore, the generated samples can be used for classification models and segmentation and detection. model, which greatly enriches the scope of retinal OCT image processing.
附图说明Description of the drawings
为了使本发明的内容更容易被清楚地理解，下面根据具体实施例并结合附图，对本发明作进一步详细的说明，其中：In order to make the content of the present invention easier to understand clearly, the present invention will be further described in detail below based on specific embodiments and in conjunction with the accompanying drawings, wherein:
图1为本发明一种实施例的基于视网膜OCT图像的CNV病灶伪造方法流程示意图；Figure 1 is a schematic flow chart of a method for forging CNV lesions based on retinal OCT images according to one embodiment of the present invention;
图2为本发明一种实施例的基于视网膜OCT图像的CNV病灶伪造装置的示意图；Figure 2 is a schematic diagram of a CNV lesion forgery device based on retinal OCT images according to an embodiment of the present invention;
图3为本发明一种实施例的CNV特征生成过程示意图；Figure 3 is a schematic diagram of the CNV feature generation process according to an embodiment of the present invention;
图4为本发明一种实施例的CNV伪造过程示意图；Figure 4 is a schematic diagram of the CNV forgery process according to an embodiment of the present invention;
图5为视网膜分层图；Figure 5 is a diagram of retinal layering;
图6为本发明一种实施例的OCT图像的采样和像素变换操作示意图。Figure 6 is a schematic diagram of OCT image sampling and pixel transformation operations according to an embodiment of the present invention.
具体实施方式Detailed ways
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明的保护范围。In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with examples. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the scope of the present invention.
下面结合附图对本发明的应用原理作详细的描述。The application principle of the present invention will be described in detail below with reference to the accompanying drawings.
实施例1Example 1
本发明实施例中提供了一种基于视网膜OCT图像的CNV病灶伪造方法，包括以下步骤：The embodiment of the present invention provides a method for forging CNV lesions based on retinal OCT images, which includes the following steps:
(1)利用真实的带CNV病灶的视网膜OCT图像，以及对应的标签，训练出CNV判别器；由于真实的带CNV病灶的视网膜OCT图像非常匮乏，为此，本发明实施例中仅需 要少量的真实的带CNV病灶的视网膜OCT图像即可；(1) Use real retinal OCT images with CNV lesions and corresponding labels to train the CNV discriminator; since real retinal OCT images with CNV lesions are very scarce, for this reason, only a small amount is needed in the embodiment of the present invention. Real retinal OCT images with CNV lesions are sufficient;
(2)基于真实的带CNV病灶的视网膜OCT图像，提取出CNV真实特征；(2) Based on real retinal OCT images with CNV lesions, extract the real features of CNV;
(3)获取CNV模拟特征；在具体实施过程中，所述CNV模拟特征的数量较大，用于最大程度的覆盖CNV特征的多样性；(3) Obtain CNV simulation features; during the specific implementation process, the number of CNV simulation features is relatively large to cover the diversity of CNV features to the greatest extent;
(4)基于所述CNV真实特征和CNV模拟特征，在不带CNV病灶的视网膜OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器；(4) Based on the CNV real features and CNV simulated features, forge CNV lesions on retinal OCT images without CNV lesions, batch generate retinal OCT images and labels with CNV lesions, and send them to the CNV discriminator;
(5)将CNV判别器输出的合格数据和真实的带CNV病灶的视网膜OCT图像送入处理OCT图像的深度学习模型进行训练，得到视网膜OCT图像处理模型；在具体实施过程中，所述深度学习模型可以是神经网络模型。(5) The qualified data output by the CNV discriminator and the real retinal OCT images with CNV lesions are sent to the deep learning model for processing OCT images for training to obtain the retinal OCT image processing model; during the specific implementation process, the deep learning The model can be a neural network model.
可见，本发明实施例中的于视网膜OCT图像的CNV病灶伪造方法，能够从少量真实的带CNV病灶的视网膜OCT图像中提取真实CNV病灶的特征，并基于提取到的真实CNV病灶的特征，在大量的不带CNV病灶的视网膜OCT图像伪造CNV病灶，最终生成高准确率的视网膜OCT图像处理模型。It can be seen that the method for forging CNV lesions in retinal OCT images in the embodiment of the present invention can extract the characteristics of real CNV lesions from a small number of real retinal OCT images with CNV lesions, and based on the extracted characteristics of real CNV lesions, A large number of retinal OCT images without CNV lesions are faked with CNV lesions, and finally a high-accuracy retinal OCT image processing model is generated.
在本发明实施例的一种具体实施方式中，由于真实的带CNV病灶的视网膜OCT图像非常匮乏，为此，所述真实的带CNV病灶的视网膜OCT图像数量小于第一阈值；为了最大程度的覆盖CNV特征的多样性，所述CNV模拟特征的数量大于第二阈值，所述第一阈值小于第二阈值。In a specific implementation of the embodiment of the present invention, since real retinal OCT images with CNV lesions are very scarce, for this reason, the number of real retinal OCT images with CNV lesions is less than the first threshold; in order to maximize To cover the diversity of CNV features, the number of CNV simulation features is greater than a second threshold, and the first threshold is less than the second threshold.
由于特定机型且有标注的带CNV病灶的视网膜OCT图像匮乏，使用传统的方式训练模型容易出现过拟合的问题。为此，在本发明实施例的一种具体实施方式中，提出基于元学习MAML算法的模型训练方式来训练获得CNV判别器。具体地，所述CNV判别器的训练方法包括以下步骤：Due to the lack of labeled retinal OCT images with CNV lesions of specific models, using traditional methods to train models is prone to overfitting problems. To this end, in a specific implementation of the embodiment of the present invention, a model training method based on the meta-learning MAML algorithm is proposed to train and obtain the CNV discriminator. Specifically, the training method of the CNV discriminator includes the following steps:
在自然数据集上使用2-ways-5query-20shot方式训练出元学习器和二分类器；Use the 2-ways-5query-20shot method to train the meta-learner and binary classifier on natural data sets;
基于所述元学习器和二分类器，在所述真实的带CNV病灶的视网膜OCT图像上继续训练，最终得到CNV判别器。Based on the meta-learner and binary classifier, training is continued on the real retinal OCT images with CNV lesions, and finally a CNV discriminator is obtained.
在本发明实施例的一种具体实施方式中，所述CNV真实特征的提取方法包括以下步骤：In a specific implementation of the embodiment of the present invention, the method for extracting true CNV features includes the following steps:
从所述真实的带CNV病灶的视网膜OCT图像上提取CNV病灶的大小、轮廓和位置特征；Extract the size, contour and location features of CNV lesions from the real retinal OCT images with CNV lesions;
对CNV病灶进行仿射变换和弹性变形，并提取仿射变换和弹性变形后的CNV病灶的大小、轮廓和位置特征，进行对CNV病灶的大小、轮廓和位置特征的扩充。在具体实施 过程中，所述仿射变换包括平移、放缩等操作，具体参见图3。Perform affine transformation and elastic deformation on CNV lesions, and extract the size, contour, and position features of CNV lesions after affine transformation and elastic deformation, and expand the size, contour, and position features of CNV lesions. In the specific implementation process, the affine transformation includes operations such as translation and scaling, see Figure 3 for details.
在本发明实施例的一种具体实施方式中，所述CNV模拟特征的获取方法包括：In a specific implementation of the embodiment of the present invention, the method for obtaining CNV simulation features includes:
基于CNV特征模拟算法，自动生成CNV病灶的大小、轮廓和位置特征；Based on the CNV feature simulation algorithm, the size, contour and location features of CNV lesions are automatically generated;
在具体实施过程中，如图3所示，所述基于CNV特征模拟算法，自动生成CNV病灶的大小、轮廓和位置特征，具体包括：首先批量生成随机大小和形状的椭圆和三角图像；然后经过透视投影变换，生成类似CNV病灶的图像；最后进行CNV病灶的大小、轮廓和位置特征提取，完成自动生成CNV病灶的大小、轮廓和位置特征。In the specific implementation process, as shown in Figure 3, the CNV feature-based simulation algorithm automatically generates the size, outline and location characteristics of CNV lesions, specifically including: first batch generating elliptical and triangular images of random sizes and shapes; and then Perspective projection transformation is used to generate images similar to CNV lesions; finally, the size, outline and position features of CNV lesions are extracted to complete the automatic generation of the size, outline and position features of CNV lesions.
本发明根据CNV真实特征进行CNV病灶伪造，可最大程度保证伪造的带CNV病灶的视网膜OCT图像与真实的带CNV病灶的视网膜OCT图像的相似性；利用CNV特征模拟算法自动、批量生成的CNV模拟特征，在保证模拟特征与真实特征相似性的同时，也最大程度覆盖CNV病灶特征的多样性。The present invention forges CNV lesions based on the true characteristics of CNV, which can ensure to the greatest extent the similarity between the forged retinal OCT images with CNV lesions and the real retinal OCT images with CNV lesions; CNV simulations are automatically and batch-generated using the CNV feature simulation algorithm. Features, while ensuring the similarity between simulated features and real features, also covers the diversity of CNV lesion features to the greatest extent.
在本发明实施例的一种具体实施方式中，如图4所示，所述带CNV病灶的视网膜OCT图像及标签的生成方法包括：In a specific implementation of the embodiment of the present invention, as shown in Figure 4, the method for generating retinal OCT images and labels with CNV lesions includes:
步骤(1)：选择一张OCT图像及其视网膜分层金标准数据，定位视网膜各层的位置，并计算视网膜各层厚度、倾斜度、亮度等特征；选择一组CNV生成特征(CNV真实特征或CNV模拟特征)；Step (1): Select an OCT image and its retinal layering gold standard data, locate the positions of each retinal layer, and calculate the thickness, tilt, brightness and other characteristics of each retinal layer; select a set of CNV generated features (CNV real features or CNV simulation features);
步骤(2)：根据步骤(1)得到的算视网膜的特征，判断该张视网膜OCT图像是否适合伪造CNV病灶，不适合则重新选择一张OCT图像及其金标准数据，适合则进行步骤(3)。Step (2): Based on the calculated retinal characteristics obtained in step (1), determine whether the retinal OCT image is suitable for forging CNV lesions. If it is not suitable, select another OCT image and its gold standard data. If it is suitable, proceed to step (3). ).
步骤(3)：根据步骤(1)计算的视网膜各层的层厚度和CNV生成特征的大小和轮廓特征，计算出适合在该张视网膜OCT图像上伪造CNV病灶的大小和轮廓，具体的计算过程包括：Step (3): Based on the layer thickness of each retinal layer calculated in step (1) and the size and outline features of the CNV generation features, calculate the size and outline suitable for forging CNV lesions on the retinal OCT image. The specific calculation process include:
视网膜各层厚度用D(d 1,d 2,d 3,d 4,d 5)表示，d 1表示脉络膜(Choroid)厚度，d 2表示色素上皮层(RPE)和威尔赫夫膜(VM)总厚度，d 3表示外节层(OSL)、连接纤毛层(CL)、外丛状层(OPL)、内核层(INL)和内丛状层(IPL)总厚度、d 4表示神经节细胞(GCL)、神经纤维层(RNFL)和玻璃体(Vitreous)的总厚度，d 5表示玻璃体(Vitreous)厚度。将CNV生成特征的大小用B(像素个数)表示，轮廓用C(X,Y)表示，O表示真实的带CNV病灶的视网膜OCT图像的分层金标准，宽和高分别用w 0和h 0表示，其中，X＝(x 1...x 2,x m)表示横坐标，Y＝(y 1...y 2,y m)表示对应的纵坐标。计算X最大的差值x max若，计算Y最大的差值y max，则适合伪造的CNV病灶的轮廓
Figure PCTCN2022094354-appb-000004
的计算公式如下： The thickness of each retinal layer is represented by D (d 1 , d 2 , d 3 , d 4 , d 5 ), d 1 represents the thickness of the choroid, and d 2 represents the pigment epithelium (RPE) and Wilhoff's membrane (VM). ) total thickness, d 3 represents the total thickness of the outer segmental layer (OSL), connecting ciliary layer (CL), outer plexiform layer (OPL), inner nuclear layer (INL) and inner plexiform layer (IPL), d 4 represents the ganglion The total thickness of cells (GCL), nerve fiber layer (RNFL) and vitreous body (Vitreous), d 5 represents the thickness of vitreous body (Vitreous). The size of the CNV generated feature is represented by B (number of pixels), the outline is represented by C (X, Y), O represents the layered gold standard of the real retinal OCT image with CNV lesions, and the width and height are represented by w 0 and w respectively. h 0 represents, where X=(x 1 ...x 2 , x m ) represents the abscissa, and Y = (y 1 ...y 2 , y m ) represents the corresponding ordinate. Calculate the maximum difference x max of
Figure PCTCN2022094354-appb-000004
The calculation formula is as follows:
Figure PCTCN2022094354-appb-000005
Figure PCTCN2022094354-appb-000005
其中，d＝d 2+d 3+d 4。若y max≥d且B<d*x max，则
Figure PCTCN2022094354-appb-000006
若y max≥d或B≥d*x max，则对C进行缩小，缩小过程包括：将图像O缩小至宽为
Figure PCTCN2022094354-appb-000007
高为
Figure PCTCN2022094354-appb-000008
e 1和e 2是0～d/4的随机整数；根据缩小后的图像O，重新提取CNV病灶的轮廓特征
Figure PCTCN2022094354-appb-000009
大小特征
Figure PCTCN2022094354-appb-000010
和位置特征，则
Figure PCTCN2022094354-appb-000011
和
Figure PCTCN2022094354-appb-000012
就是计算得到的适合伪造CNV的轮廓特征和大小特征。 Among them, d= d2 + d3 + d4 . If y max ≥ d and B < d*x max , then
Figure PCTCN2022094354-appb-000006
If y max ≥ d or B ≥ d*x max , then C is reduced. The reduction process includes: reducing the image O to a width of
Figure PCTCN2022094354-appb-000007
Gao Wei
Figure PCTCN2022094354-appb-000008
e 1 and e 2 are random integers from 0 to d/4; based on the reduced image O, the outline features of CNV lesions are re-extracted
Figure PCTCN2022094354-appb-000009
size characteristics
Figure PCTCN2022094354-appb-000010
and location characteristics, then
Figure PCTCN2022094354-appb-000011
and
Figure PCTCN2022094354-appb-000012
It is the calculated contour features and size features suitable for forging CNV.
步骤(4)：若图像O未经过缩放，则使用原始的位置特征，否则使用步骤(3)重新提取的位置特征，并结合步骤(1)定位的视网膜各层的位置，计算出适合在该张视网膜OCT图像上伪造CNV病灶的位置。计算过程具体为：Step (4): If the image O has not been scaled, use the original position features, otherwise use the position features re-extracted in step (3), combined with the positions of each retinal layer positioned in step (1), to calculate the position suitable for the Location of spurious CNV lesions on retinal OCT images. The calculation process is specifically as follows:
视网膜各层的位置特征用M[M 1,M 2,M 3]表示，M 1[W 1,H 1]表示脉络膜层与色素上皮层的分割线，M 2[W 2,H 2]表示威尔赫夫膜与外节层的分割线，M 3[W 3,H 3]表示外丛状层与内核层的分割线，其中W k＝(w 1...w n-1,w n)是表示横坐标，H k＝(h 1...h n-1,h n)表示对应的纵坐标，k＝1,2,3，n表示坐标总数；CNV生成特征的位置特征用I[(w 1,h 1),(w 2,h 2)]表示，(w 1,h 1)表示CNV生成特征的左上角坐标，(w 2,h 2)表示CNV生成特征的右下角坐标，则适合在该张视网膜OCT图像上伪造CNV病灶的位置为I(w 2,h k)，h k∈H 2作为伪造的CNV的起始位置。 The positional characteristics of each layer of the retina are represented by M[M 1 , M 2 , M 3 ], M 1 [W 1 , H 1 ] represents the dividing line between the choroidal layer and the pigment epithelium layer, and M 2 [W 2 , H 2 ] represents The dividing line between Wilhoff's membrane and the outer segmental layer, M 3 [W 3 ,H 3 ] represents the dividing line between the outer plexiform layer and the inner core layer, where W k = (w 1 ...w n-1 ,w n ) represents the abscissa, H k = (h 1 ... h n-1 , h n ) represents the corresponding ordinate, k = 1, 2, 3, n represents the total number of coordinates; the position features of CNV generated features are expressed in I[(w 1 ,h 1 ),(w 2 ,h 2 )] represents, (w 1 ,h 1 ) represents the coordinates of the upper left corner of the CNV generated feature, (w 2 ,h 2 ) represents the lower right corner of the CNV generated feature coordinates, then the position suitable for forging CNV lesions on this retinal OCT image is I(w 2 , h k ), and h k ∈ H 2 is used as the starting position of the forged CNV.
步骤(5)：根据步骤(3)和步骤(4)计算的适合伪造CNV病灶的位置、大小和轮廓，分别在该张视网膜OCT图像上及其分层金标准上进行采样和像素变换等操作，得到带CNV病灶的视网膜OCT图像及分层标签。Step (5): Based on the location, size and contour of the forged CNV lesions calculated in steps (3) and (4), perform sampling and pixel transformation operations on the retinal OCT image and its layered gold standard respectively. , obtain retinal OCT images and hierarchical labels with CNV lesions.
其中，OCT图像的采样和像素变换操作详细过程参考图6，具体包括以下步骤：Among them, the detailed process of sampling and pixel transformation operations of OCT images refers to Figure 6, which specifically includes the following steps:
首先在无CNV病灶的OCT图像上定位到适合伪造CNV的位置I(w 2,h k)，h k∈H 2，结合适合伪造CNV病灶的大小和轮廓，确定出伪造的CNV病灶； First, locate the position I(w 2 , h k ) suitable for forged CNV on the OCT image without CNV lesions, h k ∈ H 2 , and determine the forged CNV lesions based on the size and outline of the lesions suitable for forged CNV;
重新计算伪造的CNV病灶轮廓内各点的像素值，具体包括：Recalculate the pixel values of each point within the forged CNV lesion outline, including:
伪造的CNV轮廓及轮廓内的位置集合用P[P 1,P 2...P m]表示，假设需要计算伪造的CNV轮廓内的某一点P 2(w p2,h p2)的像素值；P 1(w p1,h p1)和P 3(w p3,h p3)分别为轮廓的上某两点的位置，其中w p1＝w p3＝w p2；Q(w q,h m)是外丛装层与内核分割线M 3[W 3,H 3]上的某一点，其中w q＝w p3且w q∈H 3,h q∈H 3。采样点T(w t,h t)的像素值用Q表示，位置P 2(w p2,h p2)像素值 用
Figure PCTCN2022094354-appb-000013
表示，则
Figure PCTCN2022094354-appb-000014
e为0～10之间的随机整数； The forged CNV contour and the position set within the contour are represented by P[P 1 ,P 2 ...P m ]. It is assumed that the pixel value of a certain point P 2 (w p2 ,h p2 ) within the forged CNV contour needs to be calculated; P 1 (w p1 ,h p1 ) and P 3 (w p3 ,h p3 ) are the positions of two points on the contour respectively, where w p1 =w p3 =w p2 ; Q (w q ,h m ) is the outer A certain point on the dividing line M 3 [W 3 ,H 3 ] between the bundle layer and the core, where w q =w p3 and w q ∈H 3 ,h q ∈H 3 . The pixel value of the sampling point T (w t , h t ) is represented by Q, and the pixel value of the position P 2 (w p2 , h p2 ) is represented by
Figure PCTCN2022094354-appb-000013
means, then
Figure PCTCN2022094354-appb-000014
e is a random integer between 0 and 10;
计算出采样点T(w t,h t)的位置，进而获得采样点T(w t,h t)的像素值，并根据采样点采样点T(w t,h t)的像素值计算出点P 2(w p2,h p2)的像素值，其中，采样点T(w t,h t)的位置的计算公式为： Calculate the position of the sampling point T(w t ,h t ), and then obtain the pixel value of the sampling point T(w t ,h t ), and calculate it based on the pixel value of the sampling point T(w t ,h t ) The pixel value of point P 2 (w p2 ,h p2 ), where the calculation formula for the position of sampling point T (w t ,h t ) is:
w t＝w p2±σ,σ是(-3,3)之间的随机整数 w t =w p2 ±σ,σ is a random integer between (-3,3)
Figure PCTCN2022094354-appb-000015
Figure PCTCN2022094354-appb-000015
由于位置和像素值是对应的，当采样点T(w t,h t)的位置被计算出来后，其像素值可以基于key-value获得。 Since the position and pixel value are corresponding, when the position of the sampling point T (w t , h t ) is calculated, its pixel value can be obtained based on key-value.
分层金标准的采样和像素变换操作和OCT图像的保持一致。The sampling and pixel transformation operations of the layered gold standard are consistent with those of OCT images.
在本发明实施例的一种具体实施方式中，生成的标签为分类标签，所述视网膜OCT图像处理模型为分类模型。在本发明实施例的其他实施方式中，生成的标签还可以是分割标签或检测标签，所述视网膜OCT图像处理模型还可以是分割模型或检测模型。所述分类标签、分割标签或检测标签均可以通过前述分层特征通过现有技术转换而成。In a specific implementation of the embodiment of the present invention, the generated labels are classification labels, and the retinal OCT image processing model is a classification model. In other implementations of the embodiments of the present invention, the generated labels may also be segmentation labels or detection labels, and the retinal OCT image processing model may also be a segmentation model or a detection model. The classification tags, segmentation tags or detection tags can be converted by using the aforementioned hierarchical features through existing technologies.
下面结合一具体实施方式，对本发明实施例中的CNV病灶伪造方法进行详细说明。The CNV lesion forgery method in the embodiment of the present invention will be described in detail below with reference to a specific implementation.
针对带标注的且包含CNV病灶的视网膜OCT图像稀缺、标注成本高、模型容易过拟合等问题，本发明提供一种视网膜OCT图像的CNV病灶伪造方法，如图1所示，可以分为三个步骤：Aiming at the scarcity of annotated retinal OCT images containing CNV lesions, high annotation costs, and easy over-fitting of models, the present invention provides a CNV lesion forgery method for retinal OCT images, as shown in Figure 1, which can be divided into three types. steps:
步骤一、CNV判别器训练、CNV特征提取和自动生成Step 1. CNV discriminator training, CNV feature extraction and automatic generation
1.1、利用CNV病灶特征提取算法，从少量的真实的带CNV病灶的视网膜OCT图像上，提取CNV病灶的大小、轮廓和位置特征，并对真实的CNV病灶进行仿射变换、弹性变形，对CNV病灶的大小、轮廓和位置特征进行扩充。1.1. Use the CNV lesion feature extraction algorithm to extract the size, contour and location features of CNV lesions from a small number of real retinal OCT images with CNV lesions, and perform affine transformation and elastic deformation on the real CNV lesions to perform CNV The size, contour and location characteristics of the lesions are expanded.
1.2、利用CNV特征模拟算法，自动生成CNV的大小、轮廓、位置等特征；1.2. Use the CNV feature simulation algorithm to automatically generate the size, outline, position and other features of CNV;
1.3、由于特定机型且有标注的带CNV病灶的视网膜OCT图像匮乏，使用传统的方式训练模型容易出现过拟合的问题，因此本发明提供一种基于元学习MAML算法的模型训练方式：首先在自然数据集上使用2-ways-5query-20shot方式训练出元学习器和二分类器，然后基于此元学习器和二分类器，在少量带CNV病灶的视网膜OCT图像上继续训练，最终得到CNV判别器。1.3. Due to the lack of labeled retinal OCT images with CNV lesions of specific models, using traditional methods to train models is prone to over-fitting problems. Therefore, the present invention provides a model training method based on the meta-learning MAML algorithm: First, The meta-learner and binary classifier are trained using the 2-ways-5query-20shot method on the natural data set, and then based on this meta-learner and binary classifier, continue training on a small number of retinal OCT images with CNV lesions, and finally get CNV discriminator.
步骤二、CNV病灶伪造和生成样本筛选Step 2: Forgery of CNV lesions and screening of generated samples
2.1、根据步骤一种提取的CNV真实特征和算法自动生成的CNV模拟特征，在大量不 带CNV病灶的视网膜OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签。2.1. Based on the CNV real features extracted in step 1 and the CNV simulation features automatically generated by the algorithm, forge CNV lesions on a large number of retinal OCT images without CNV lesions, and batch generate retinal OCT images and labels with CNV lesions.
2.2、使用步骤一训练出的CNV判别器，判断生成的带CNV的OCT图像及标签数据是否合格。不合格则返回步骤(1)，销毁并重新生成，合格则进入步骤三；2.2. Use the CNV discriminator trained in step 1 to determine whether the generated OCT images and label data with CNV are qualified. If it is unqualified, return to step (1), destroy and regenerate; if it is qualified, go to step three;
步骤三、模型训练和验证Step 3. Model training and verification
3.1、使用步骤二中批量生成的带CNV病灶的视网膜OCT图像和标签，送入处理OCT图像的深度学习模型进行训练，生成视网膜OCT图像处理模型，以提高模型的准确率并避免模型过拟合；所述视网膜OCT图像处理模型可以用于进行视网膜OCT图像分类、视网膜OCT疾病分割和视网膜OCT疾病检测。3.1. Use the retinal OCT images and labels with CNV lesions generated in batches in step 2, send them to the deep learning model that processes OCT images for training, and generate a retinal OCT image processing model to improve the accuracy of the model and avoid model overfitting. ; The retinal OCT image processing model can be used for retinal OCT image classification, retinal OCT disease segmentation and retinal OCT disease detection.
实施例2Example 2
基于与实施例1相同的发明构思，本发明实施例中提供了一种基于视网膜OCT图像的CNV病灶伪造装置，包括：Based on the same inventive concept as in Embodiment 1, the embodiment of the present invention provides a device for forging CNV lesions based on retinal OCT images, including:
训练模块，用于利用所述真实的带CNV病灶的视网膜OCT图像，以及对应的标签，训练出CNV判别器；A training module used to train a CNV discriminator using the real retinal OCT images with CNV lesions and corresponding labels;
提取模块，用于基于所述真实的带CNV病灶的视网膜OCT图像，提取出CNV真实特征；An extraction module, configured to extract real features of CNV based on the real retinal OCT image with CNV lesions;
获取模块，用于获取CNV模拟特征；Acquisition module, used to obtain CNV simulation features;
伪造模块，用于基于所述CNV真实特征和CNV模拟特征，在不带CNV病灶的视网膜OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器；A forgery module, used to forge CNV lesions on retinal OCT images without CNV lesions based on the CNV real features and CNV simulated features, batch generate retinal OCT images and labels with CNV lesions, and send them to the CNV discriminator ;
模型训练模块，用于将CNV判别器输出的合格数据和真实的带CNV病灶的视网膜OCT图像送入处理OCT图像的深度学习模型进行训练，得到视网膜OCT图像处理模型。The model training module is used to send qualified data output by the CNV discriminator and real retinal OCT images with CNV lesions into the deep learning model that processes OCT images for training, and obtain a retinal OCT image processing model.
可见，本发明实施例中的于视网膜OCT图像的CNV病灶伪造装置，能够从少量真实的带CNV病灶的视网膜OCT图像中提取真实CNV病灶的特征，并基于提取到的真实CNV病灶的特征，在大量的不带CNV病灶的视网膜OCT图像伪造CNV病灶，最终生成高准确率的视网膜OCT图像处理模型。It can be seen that the CNV lesion forgery device for retinal OCT images in the embodiment of the present invention can extract the characteristics of real CNV lesions from a small number of real retinal OCT images with CNV lesions, and based on the extracted characteristics of real CNV lesions, A large number of retinal OCT images without CNV lesions are faked with CNV lesions, and finally a high-accuracy retinal OCT image processing model is generated.
在本发明实施例的一种具体实施方式中，由于真实的带CNV病灶的视网膜OCT图像非常匮乏，为此，所述真实的带CNV病灶的视网膜OCT图像数量小于第一阈值；为了最大程度的覆盖CNV特征的多样性，所述CNV模拟特征的数量大于第二阈值，所述第一阈值小于第二阈值。In a specific implementation of the embodiment of the present invention, since real retinal OCT images with CNV lesions are very scarce, for this reason, the number of real retinal OCT images with CNV lesions is less than the first threshold; in order to maximize To cover the diversity of CNV features, the number of CNV simulation features is greater than a second threshold, and the first threshold is less than the second threshold.
由于特定机型且有标注的带CNV病灶的视网膜OCT图像匮乏，使用传统的方式训练 模型容易出现过拟合的问题。为此，在本发明实施例的一种具体实施方式中，提出基于元学习MAML算法的模型训练方式来训练获得CNV判别器。具体地，所述CNV判别器的训练方法包括以下步骤：Due to the lack of labeled retinal OCT images with CNV lesions of specific models, training models using traditional methods is prone to overfitting problems. To this end, in a specific implementation of the embodiment of the present invention, a model training method based on the meta-learning MAML algorithm is proposed to train and obtain the CNV discriminator. Specifically, the training method of the CNV discriminator includes the following steps:
在自然数据集上使用2-ways-5query-20shot方式训练出元学习器和二分类器；Use the 2-ways-5query-20shot method to train the meta-learner and binary classifier on natural data sets;
基于所述元学习器和二分类器，在所述真实的带CNV病灶的视网膜OCT图像上继续训练，最终得到CNV判别器。Based on the meta-learner and binary classifier, training is continued on the real retinal OCT images with CNV lesions, and finally a CNV discriminator is obtained.
在本发明实施例的一种具体实施方式中，所述CNV真实特征的提取方法包括以下步骤：In a specific implementation of the embodiment of the present invention, the method for extracting true CNV features includes the following steps:
从所述真实的带CNV病灶的视网膜OCT图像上提取CNV病灶的大小、轮廓和位置特征；Extract the size, contour and location features of CNV lesions from the real retinal OCT images with CNV lesions;
对CNV病灶进行仿射变换和弹性变形，并提取仿射变换和弹性变形后的CNV病灶的大小、轮廓和位置特征，进行对CNV病灶的大小、轮廓和位置特征的扩充。在具体实施过程中，所述仿射变换包括平移、放缩等操作，具体参见图3。Perform affine transformation and elastic deformation on CNV lesions, and extract the size, contour, and position features of CNV lesions after affine transformation and elastic deformation, and expand the size, contour, and position features of CNV lesions. In the specific implementation process, the affine transformation includes operations such as translation and scaling, see Figure 3 for details.
在本发明实施例的一种具体实施方式中，所述CNV模拟特征的获取方法包括：In a specific implementation of the embodiment of the present invention, the method for obtaining CNV simulation features includes:
基于CNV特征模拟算法，自动生成CNV病灶的大小、轮廓和位置特征；Based on the CNV feature simulation algorithm, the size, contour and location features of CNV lesions are automatically generated;
在具体实施过程中，如图3所示，所述基于CNV特征模拟算法，自动生成CNV病灶的大小、轮廓和位置特征，具体包括：首先批量生成随机大小和形状的椭圆和三角图像；然后经过透视投影变换，生成类似CNV病灶的图像；最后进行CNV病灶的大小、轮廓和位置特征提取，完成自动生成CNV病灶的大小、轮廓和位置特征。In the specific implementation process, as shown in Figure 3, the CNV feature-based simulation algorithm automatically generates the size, outline and location characteristics of CNV lesions, specifically including: first batch generating elliptical and triangular images of random sizes and shapes; and then Perspective projection transformation is used to generate images similar to CNV lesions; finally, the size, outline and position features of CNV lesions are extracted to complete the automatic generation of the size, outline and position features of CNV lesions.
本发明根据CNV真实特征进行CNV病灶伪造，可最大程度保证伪造的带CNV病灶的视网膜OCT图像与真实的带CNV病灶的视网膜OCT图像的相似性；利用CNV特征模拟算法自动、批量生成的CNV模拟特征，在保证模拟特征与真实特征相似性的同时，也最大程度覆盖CNV病灶特征的多样性。The present invention forges CNV lesions based on the true characteristics of CNV, which can ensure to the greatest extent the similarity between the forged retinal OCT images with CNV lesions and the real retinal OCT images with CNV lesions; CNV simulations are automatically and batch-generated using the CNV feature simulation algorithm. Features, while ensuring the similarity between simulated features and real features, also covers the diversity of CNV lesion features to the greatest extent.
在本发明实施例的一种具体实施方式中，如图4所示，所述带CNV病灶的视网膜OCT图像及标签的生成方法包括：In a specific implementation of the embodiment of the present invention, as shown in Figure 4, the method for generating retinal OCT images and labels with CNV lesions includes:
步骤(1)：选择一张OCT图像及其视网膜分层金标准数据，定位视网膜各层的位置，并计算视网膜各层厚度、倾斜度、亮度等特征；选择一组CNV生成特征(CNV真实特征或CNV模拟特征)；Step (1): Select an OCT image and its retinal layering gold standard data, locate the positions of each retinal layer, and calculate the thickness, tilt, brightness and other characteristics of each retinal layer; select a set of CNV generated features (CNV real features or CNV simulation features);
步骤(2)：根据步骤(1)得到的算视网膜的特征，判断该张视网膜OCT图像是否适合伪造CNV病灶，不适合则重新选择一张OCT图像及其金标准数据，适合则进行步骤(3)。Step (2): Based on the calculated retinal characteristics obtained in step (1), determine whether the retinal OCT image is suitable for forging CNV lesions. If it is not suitable, select another OCT image and its gold standard data. If it is suitable, proceed to step (3). ).
步骤(3)：根据步骤(1)计算的视网膜各层的层厚度和CNV生成特征的大小和轮廓特征，计算出适合在该张视网膜OCT图像上伪造病灶的大小和轮廓，具体的计算过程包括：Step (3): Based on the layer thickness of each retinal layer calculated in step (1) and the size and outline features of the CNV generation features, calculate the size and outline suitable for forging the lesion on the retinal OCT image. The specific calculation process includes :
如图5所示，视网膜各层厚度用D(d 1,d 2,d 3,d 4,d 5)表示，d 1表示脉络膜(Choroid)厚度，d 2表示色素上皮层(RPE)和威尔赫夫膜(VM)总厚度，d 3表示外节层(OSL)、连接纤毛层(CL)、外丛状层(OPL)、内核层(INL)和内丛状层(IPL)总厚度、d 4表示神经节细胞(GCL)、神经纤维层(RNFL)和玻璃体(Vitreous)的总厚度，d 5表示玻璃体(Vitreous)厚度。将CNV生成特征的大小用B(像素个数)表示，轮廓用C(X,Y)表示，O表示真实的带CNV病灶的视网膜OCT图像的分层金标准，宽和高分别用w 0和h 0表示，其中，X＝(x 1...x 2,x m)表示横坐标，Y＝(y 1...y 2,y m)表示对应的纵坐标。计算X最大的差值x max若，计算Y最大的差值y max，则适合伪造的CNV的轮廓
Figure PCTCN2022094354-appb-000016
的计算公式如下： As shown in Figure 5, the thickness of each layer of the retina is represented by D (d 1 , d 2 , d 3 , d 4 , d 5 ), d 1 represents the choroid (Choroid) thickness, d 2 represents the pigment epithelium (RPE) and the thickness of the retina. The total thickness of Erhöf's membrane (VM), d 3 represents the total thickness of the outer segment layer (OSL), connecting ciliary layer (CL), outer plexiform layer (OPL), inner core layer (INL) and inner plexiform layer (IPL) , d 4 represents the total thickness of ganglion cells (GCL), nerve fiber layer (RNFL) and vitreous body (Vitreous), and d 5 represents the thickness of vitreous body (Vitreous). The size of the CNV generated feature is represented by B (number of pixels), the outline is represented by C (X, Y), O represents the layered gold standard of the real retinal OCT image with CNV lesions, and the width and height are represented by w 0 and w respectively. h 0 represents, where X=(x 1 ...x 2 , x m ) represents the abscissa, and Y = (y 1 ...y 2 , y m ) represents the corresponding ordinate. Calculate the maximum difference x max of
Figure PCTCN2022094354-appb-000016
The calculation formula is as follows:
Figure PCTCN2022094354-appb-000017
Figure PCTCN2022094354-appb-000017
其中，d＝d 2+d 3+d 4。若y max≥d且B<d*x max，则
Figure PCTCN2022094354-appb-000018
若y max≥d或B≥d*x max，则对C进行缩小，缩小方式包括： Among them, d= d2 + d3 + d4 . If y max ≥ d and B < d*x max , then
Figure PCTCN2022094354-appb-000018
If y max ≥ d or B ≥ d*x max , then C will be reduced. The reduction methods include:
(1)将图像O缩小至宽为
Figure PCTCN2022094354-appb-000019
高为
Figure PCTCN2022094354-appb-000020
e 1和e 2是0～d/4的随机整数。 (1) Reduce the image O to a width of
Figure PCTCN2022094354-appb-000019
Gao Wei
Figure PCTCN2022094354-appb-000020
e 1 and e 2 are random integers from 0 to d/4.
(2)根据缩小后的图像O，重新提取CNV病灶的轮廓特征
Figure PCTCN2022094354-appb-000021
大小特征
Figure PCTCN2022094354-appb-000022
和位置特征，则
Figure PCTCN2022094354-appb-000023
和
Figure PCTCN2022094354-appb-000024
就是计算得到的适合伪造CNV的轮廓特征和大小特征。 (2) Based on the reduced image O, re-extract the outline features of CNV lesions
Figure PCTCN2022094354-appb-000021
size characteristics
Figure PCTCN2022094354-appb-000022
and location characteristics, then
Figure PCTCN2022094354-appb-000023
and
Figure PCTCN2022094354-appb-000024
It is the calculated contour features and size features suitable for forging CNV.
步骤(4)：若图像O未经过缩放，则使用原始的位置特征，否则使用步骤(3)重新提取的位置特征，并结合步骤(1)定位的视网膜各层的位置，计算出适合在该张视网膜OCT图像上伪造CNV病灶的位置。计算过程具体为：Step (4): If the image O has not been scaled, use the original position features, otherwise use the position features re-extracted in step (3), combined with the positions of each retinal layer positioned in step (1), to calculate the position suitable for the Location of spurious CNV lesions on retinal OCT images. The calculation process is specifically as follows:
视网膜各层的位置特征用M[M 1,M 2,M 3]表示，M 1[W 1,H 1]表示脉络膜层与色素上皮层的分割线，M 2[W 2,H 2]表示威尔赫夫膜与外节层的分割线，M 3[W 3,H 3]表示外丛状层与内核层的分割线，其中W k＝(w 1...w n-1,w n)是表示横坐标，H k＝(h 1...h n-1,h n)表示对应的纵坐标；CNV生成特征的位置特征I[(w 1,h 1),(w 2,h 2)]用表示，(w 1,h 1)表示CNV左上角坐标，(w 2,h 2)表示CNV右下角坐标；则适合在该张视网膜OCT图像上伪造CNV病灶的位置为
Figure PCTCN2022094354-appb-000025
h k∈H 2作为伪造的CNV的起始位置。 The positional characteristics of each layer of the retina are represented by M[M 1 , M 2 , M 3 ], M 1 [W 1 , H 1 ] represents the dividing line between the choroidal layer and the pigment epithelium layer, and M 2 [W 2 , H 2 ] represents The dividing line between Wilhoff's membrane and the outer segmental layer, M 3 [W 3 ,H 3 ] represents the dividing line between the outer plexiform layer and the inner core layer, where W k = (w 1 ...w n-1 ,w n ) represents the abscissa, H k = (h 1 ... h n-1 , h n ) represents the corresponding ordinate; the position feature I [(w 1 , h 1 ), (w 2 , ) of the CNV generated feature h 2 ) ] expressed by
Figure PCTCN2022094354-appb-000025
h k ∈H 2 serves as the starting position of the forged CNV.
步骤(5)根据步骤(3)和步骤(4)计算的适合伪造CNV病灶的位置、大小和轮廓，分别在该张视网膜OCT图像上及其分层金标准上进行采样和像素变换等操作，得到带CNV病灶的视网膜OCT图像及分层标签。Step (5) Based on the location, size and contour of the suitable forged CNV lesion calculated in steps (3) and (4), perform operations such as sampling and pixel transformation on the retinal OCT image and its layered gold standard respectively. Retinal OCT images with CNV lesions and hierarchical labels were obtained.
OCT图像的采样和像素变换操作详细过程参考图6，具体包括以下步骤：The detailed process of sampling and pixel transformation operations of OCT images refers to Figure 6, which specifically includes the following steps:
首先在无CNV病灶的OCT图像上定位到适合伪造CNV的位置I(w 2,h k)，h k∈H 2，结合适合伪造病灶的大小和轮廓，确定出伪造的CNV病灶； First, locate the position I(w 2 , h k ) suitable for forged CNV on the OCT image without CNV lesions, h k ∈ H 2 , and determine the forged CNV lesions based on the size and contour suitable for the forged lesions;
重新计算CNV病灶轮廓内各点的像素值，具体包括：CNV轮廓及轮廓内的位置集合用P[P 1,P 2...P m]表示，假设需要计算CNV轮廓内的某一点P 2(w p2,h p2)的像素值；P 1(w p1,h p1)和P 3(w p3,h p3)分别为轮廓的上某两点的位置，其中w p1＝w p3＝w p2；Q(w q,h m)是外丛装层与内核分割线M 3[W 3,H 3]上的某一点，其中w q＝w p3且w q∈H 3,h q∈H 3。采样点T(w t,h t)的像素值用Q表示，位置P 2(w p2,h p2)像素值用
Figure PCTCN2022094354-appb-000026
表示，则
Figure PCTCN2022094354-appb-000027
e为0～10之间的随机整数； Recalculate the pixel values of each point within the CNV lesion outline, including: the CNV outline and the position set within the outline are represented by P [P 1 , P 2 ...P m ]. It is assumed that a certain point P 2 within the CNV outline needs to be calculated. The pixel value of (w p2 , h p2 ); P 1 (w p1 , h p1 ) and P 3 (w p3 , h p3 ) are the positions of two points on the outline respectively, where w p1 = w p3 = w p2 ; Q(w q ,h m ) is a point on the dividing line M 3 [W 3 ,H 3 ] between the outer cladding layer and the inner core, where w q =w p3 and w q ∈H 3 ,h q ∈H 3 . The pixel value of the sampling point T (w t , h t ) is represented by Q, and the pixel value of the position P 2 (w p2 , h p2 ) is represented by
Figure PCTCN2022094354-appb-000026
means, then
Figure PCTCN2022094354-appb-000027
e is a random integer between 0 and 10;
计算出采样点T(w t,h t)的位置，进而获得采样点T(w t,h t)的像素值，并根据采样点采样点T(w t,h t)的像素值计算出点P 2(w p2,h p2)的像素值，其中，采样点T(w t,h t)的位置的计算公式为： Calculate the position of the sampling point T(w t ,h t ), and then obtain the pixel value of the sampling point T(w t ,h t ), and calculate it based on the pixel value of the sampling point T(w t ,h t ) The pixel value of point P 2 (w p2 ,h p2 ), where the calculation formula for the position of sampling point T (w t ,h t ) is:
w t＝w p2±σ,σ是(-3,3)之间的随机整数 w t =w p2 ±σ,σ is a random integer between (-3,3)
Figure PCTCN2022094354-appb-000028
Figure PCTCN2022094354-appb-000028
由于位置和像素值是对应的，当采样点T(w t,h t)的位置被计算出来后，其像素值可以基于key-value获得。 Since the position and pixel value are corresponding, when the position of the sampling point T (w t ,h t ) is calculated, its pixel value can be obtained based on key-value.
分层金标准的采样和像素变换操作和OCT图像的保持一致。The sampling and pixel transformation operations of the layered gold standard are consistent with those of OCT images.
在本发明实施例的一种具体实施方式中，生成的标签为分类标签，所述视网膜OCT图像处理模型为分类模型。在本发明实施例的其他实施方式中，生成的标签还可以是分割标签或检测标签，所述视网膜OCT图像处理模型还可以是分割模型或检测模型。In a specific implementation of the embodiment of the present invention, the generated labels are classification labels, and the retinal OCT image processing model is a classification model. In other implementations of the embodiments of the present invention, the generated labels may also be segmentation labels or detection labels, and the retinal OCT image processing model may also be a segmentation model or a detection model.
实施例3Example 3
本发明实施例中提供了一种基于视网膜OCT图像的CNV病灶伪造装置，如图2所示，包括：CNV特征生成模块、CNV判别器、CNV病灶伪造模块和模型训练模块；The embodiment of the present invention provides a CNV lesion forgery device based on retinal OCT images, as shown in Figure 2, including: a CNV feature generation module, a CNV discriminator, a CNV lesion forgery module and a model training module;
所述CNV特征生成模块包括CNV特征提取器、CNV特征模拟器和特征扩充器，所 述CNV特征提取器从真实的带CNV病灶的视网膜OCT图像中提取CNV病灶的CNV真实特征，然后输入特征扩充器进行仿射和弹性变换，批量扩充CNV病灶的特征；所述CNV特征模拟器批量生成CNV模拟特征；The CNV feature generation module includes a CNV feature extractor, a CNV feature simulator and a feature expander. The CNV feature extractor extracts the CNV real features of CNV lesions from real retinal OCT images with CNV lesions, and then inputs the feature expansion The CNV feature simulator performs affine and elastic transformation to batch-expand the features of CNV lesions; the CNV feature simulator batch-generates CNV simulation features;
所述CNV判别器是在真实的带CNV病灶的视网膜OCT图像上使用元学习的方式训练获得；The CNV discriminator is trained using meta-learning on real retinal OCT images with CNV lesions;
所述CNV病灶伪造模块使用CNV特征生成模块输出的CNV真实特征和CNV模拟特征，在大量无CNV病灶的OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器进行判断；The CNV lesion forgery module uses the CNV real features and CNV simulated features output by the CNV feature generation module to forge CNV lesions on a large number of OCT images without CNV lesions, batch generate retinal OCT images and labels with CNV lesions, and send them to the institute. The CNV discriminator is used to judge;
所述模型训练模块接收CNV判别器输出的合格的数据和真实的带CNV病灶的视网膜OCT图像，并进行训练，得到视网膜OCT图像处理模型。The model training module receives qualified data output by the CNV discriminator and real retinal OCT images with CNV lesions, and performs training to obtain a retinal OCT image processing model.
可见，本发明实施例中的于视网膜OCT图像的CNV病灶伪造装置，能够从少量真实的带CNV病灶的视网膜OCT图像中提取真实CNV病灶的特征，并基于提取到的真实CNV病灶的特征，在大量的不带CNV病灶的视网膜OCT图像伪造CNV病灶，最终生成高准确率的视网膜OCT图像处理模型。It can be seen that the CNV lesion forgery device for retinal OCT images in the embodiment of the present invention can extract the characteristics of real CNV lesions from a small number of real retinal OCT images with CNV lesions, and based on the extracted characteristics of real CNV lesions, A large number of retinal OCT images without CNV lesions are faked with CNV lesions, and finally a high-accuracy retinal OCT image processing model is generated.
在本发明实施例的一种具体实施方式中，由于真实的带CNV病灶的视网膜OCT图像非常匮乏，为此，所述真实的带CNV病灶的视网膜OCT图像数量小于第一阈值；为了最大程度的覆盖CNV特征的多样性，所述CNV模拟特征的数量大于第二阈值，所述第一阈值小于第二阈值。In a specific implementation of the embodiment of the present invention, since real retinal OCT images with CNV lesions are very scarce, for this reason, the number of real retinal OCT images with CNV lesions is less than the first threshold; in order to maximize To cover the diversity of CNV features, the number of CNV simulation features is greater than a second threshold, and the first threshold is less than the second threshold.
由于特定机型且有标注的带CNV病灶的视网膜OCT图像匮乏，使用传统的方式训练模型容易出现过拟合的问题。为此，在本发明实施例的一种具体实施方式中，提出基于元学习MAML算法的模型训练方式来训练获得CNV判别器。具体地，所述CNV判别器的训练方法包括以下步骤：Due to the lack of labeled retinal OCT images with CNV lesions of specific models, using traditional methods to train models is prone to overfitting problems. To this end, in a specific implementation of the embodiment of the present invention, a model training method based on the meta-learning MAML algorithm is proposed to train and obtain the CNV discriminator. Specifically, the training method of the CNV discriminator includes the following steps:
在自然数据集上使用2-ways-5query-20shot方式训练出元学习器和二分类器；Use the 2-ways-5query-20shot method to train the meta-learner and binary classifier on natural data sets;
基于所述元学习器和二分类器，在所述真实的带CNV病灶的视网膜OCT图像上继续训练，最终得到CNV判别器。Based on the meta-learner and binary classifier, training is continued on the real retinal OCT images with CNV lesions, and finally a CNV discriminator is obtained.
在本发明实施例的一种具体实施方式中，所述CNV真实特征的提取方法包括以下步骤：In a specific implementation of the embodiment of the present invention, the method for extracting true CNV features includes the following steps:
从所述真实的带CNV病灶的视网膜OCT图像上提取CNV病灶的大小、轮廓和位置特征；Extract the size, contour and location features of CNV lesions from the real retinal OCT images with CNV lesions;
对CNV病灶进行仿射变换和弹性变形，提取CNV病灶的大小、轮廓和位置特征，进行CNV病灶的大小、轮廓和位置特征扩充。Perform affine transformation and elastic deformation on CNV lesions, extract the size, contour and position features of CNV lesions, and expand the size, contour and position features of CNV lesions.
在本发明实施例的一种具体实施方式中，所述CNV模拟特征的获取方法包括：In a specific implementation of the embodiment of the present invention, the method for obtaining CNV simulation features includes:
基于CNV特征模拟算法，自动生成CNV病灶的大小、轮廓和位置特征。Based on the CNV feature simulation algorithm, the size, contour and location features of CNV lesions are automatically generated.
在本发明实施例的一种具体实施方式中，如图4所示，所述带CNV病灶的视网膜OCT图像及标签的生成方法包括：In a specific implementation of the embodiment of the present invention, as shown in Figure 4, the method for generating retinal OCT images and labels with CNV lesions includes:
步骤(1)：选择一张OCT图像及其视网膜分层金标准数据，定位视网膜各层的位置，并计算视网膜各层厚度、倾斜度、亮度等特征；选择一组CNV生成特征(CNV真实特征或CNV模拟特征)；Step (1): Select an OCT image and its retinal layering gold standard data, locate the positions of each retinal layer, and calculate the thickness, tilt, brightness and other characteristics of each retinal layer; select a set of CNV generated features (CNV real features or CNV simulation features);
步骤(2)：根据步骤(1)得到的算视网膜的特征，判断该张视网膜OCT图像是否适合伪造CNV病灶，不适合则重新选择一张OCT图像及其金标准数据，适合则进行步骤(3)。Step (2): Based on the calculated retinal characteristics obtained in step (1), determine whether the retinal OCT image is suitable for forging CNV lesions. If it is not suitable, select another OCT image and its gold standard data. If it is suitable, proceed to step (3). ).
步骤(3)：根据步骤(1)计算的视网膜各层的层厚度和CNV生成特征的大小和轮廓特征，计算出适合在该张视网膜OCT图像上伪造病灶的大小和轮廓，具体的计算过程包括：Step (3): Based on the layer thickness of each retinal layer calculated in step (1) and the size and outline features of the CNV generation features, calculate the size and outline suitable for forging the lesion on the retinal OCT image. The specific calculation process includes :
如图5所示，视网膜各层厚度用D(d 1,d 2,d 3,d 4,d 5)表示，d 1表示脉络膜(Choroid)厚度，d 2表示色素上皮层(RPE)和威尔赫夫膜(VM)总厚度，d 3表示外节层(OSL)、连接纤毛层(CL)、外丛状层(OPL)、内核层(INL)和内丛状层(IPL)总厚度、d 4表示神经节细胞(GCL)、神经纤维层(RNFL)和玻璃体(Vitreous)的总厚度，d 5表示玻璃体(Vitreous)厚度。将CNV生成特征的大小用B(像素个数)表示，轮廓用C(X,Y)表示，O表示真实的带CNV病灶的视网膜OCT图像的分层金标准，宽和高分别用w 0和h 0表示，其中，X＝(x 1...x 2,x m)表示横坐标，Y＝(y 1...y 2,y m)表示对应的纵坐标。计算X最大的差值x max若，计算Y最大的差值y max，则适合伪造的CNV的轮廓
Figure PCTCN2022094354-appb-000029
的计算公式如下： As shown in Figure 5, the thickness of each layer of the retina is represented by D (d 1 , d 2 , d 3 , d 4 , d 5 ), d 1 represents the choroid (Choroid) thickness, d 2 represents the pigment epithelium (RPE) and the thickness of the retina. The total thickness of Erhöf's membrane (VM), d 3 represents the total thickness of the outer segment layer (OSL), connecting ciliary layer (CL), outer plexiform layer (OPL), inner core layer (INL) and inner plexiform layer (IPL) , d 4 represents the total thickness of ganglion cells (GCL), nerve fiber layer (RNFL) and vitreous body (Vitreous), and d 5 represents the thickness of vitreous body (Vitreous). The size of the CNV generated feature is represented by B (number of pixels), the outline is represented by C (X, Y), O represents the layered gold standard of the real retinal OCT image with CNV lesions, and the width and height are represented by w 0 and w respectively. h 0 represents, where X=(x 1 ...x 2 , x m ) represents the abscissa, and Y = (y 1 ...y 2 , y m ) represents the corresponding ordinate. Calculate the maximum difference x max of
Figure PCTCN2022094354-appb-000029
The calculation formula is as follows:
Figure PCTCN2022094354-appb-000030
Figure PCTCN2022094354-appb-000030
其中，d＝d 2+d 3+d 4。若y max≥d且B<d*x max，则
Figure PCTCN2022094354-appb-000031
若y max≥d或B≥d*x max，则对C进行缩小，缩小方式包括： Among them, d= d2 + d3 + d4 . If y max ≥ d and B < d*x max , then
Figure PCTCN2022094354-appb-000031
If y max ≥ d or B ≥ d*x max , then C will be reduced. The reduction methods include:
(1)将图像O缩小至宽为
Figure PCTCN2022094354-appb-000032
高为
Figure PCTCN2022094354-appb-000033
e 1和e 2是0～d/4的随机整数。 (1) Reduce the image O to a width of
Figure PCTCN2022094354-appb-000032
Gao Wei
Figure PCTCN2022094354-appb-000033
e 1 and e 2 are random integers from 0 to d/4.
(2)根据缩小后的图像O，重新提取CNV病灶的轮廓特征
Figure PCTCN2022094354-appb-000034
大小特征
Figure PCTCN2022094354-appb-000035
和位置特征， 则
Figure PCTCN2022094354-appb-000036
和
Figure PCTCN2022094354-appb-000037
就是计算得到的适合伪造CNV的轮廓特征和大小特征。 (2) Based on the reduced image O, re-extract the outline features of CNV lesions
Figure PCTCN2022094354-appb-000034
size characteristics
Figure PCTCN2022094354-appb-000035
and location features, then
Figure PCTCN2022094354-appb-000036
and
Figure PCTCN2022094354-appb-000037
It is the calculated contour features and size features suitable for forging CNV.
步骤(4)：若图像O未经过缩放，则使用原始的位置特征，否则使用步骤(3)重新提取的位置特征，并结合步骤(1)定位的视网膜各层的位置，计算出适合在该张视网膜OCT图像上伪造CNV病灶的位置。计算过程具体为：Step (4): If the image O has not been scaled, use the original position features, otherwise use the position features re-extracted in step (3), combined with the positions of each retinal layer positioned in step (1), to calculate the position suitable for the Location of spurious CNV lesions on retinal OCT images. The calculation process is specifically as follows:
视网膜各层的位置特征用M[M 1,M 2,M 3]表示，M 1[W 1,H 1]表示脉络膜层与色素上皮层的分割线，M 2[W 2,H 2]表示威尔赫夫膜与外节层的分割线，M 3[W 3,H 3]表示外丛状层与内核层的分割线，其中W k＝(w 1...w n-1,w n)是表示横坐标，H k＝(h 1...h n-1,h n)表示对应的纵坐标；CNV生成特征的位置特征I[(w 1,h 1),(w 2,h 2)]用表示，(w 1,h 1)表示CNV左上角坐标，(w 2,h 2)表示CNV右下角坐标；则适合在该张视网膜OCT图像上伪造CNV病灶的位置为
Figure PCTCN2022094354-appb-000038
h k∈H 2作为伪造的CNV的起始位置。 The positional characteristics of each layer of the retina are represented by M[M 1 , M 2 , M 3 ], M 1 [W 1 , H 1 ] represents the dividing line between the choroidal layer and the pigment epithelium layer, and M 2 [W 2 , H 2 ] represents The dividing line between Wilhoff's membrane and the outer segmental layer, M 3 [W 3 ,H 3 ] represents the dividing line between the outer plexiform layer and the inner core layer, where W k = (w 1 ...w n-1 ,w n ) represents the abscissa, H k = (h 1 ... h n-1 , h n ) represents the corresponding ordinate; the position feature I [(w 1 , h 1 ), (w 2 , ) of the CNV generated feature h 2 ) ] expressed by
Figure PCTCN2022094354-appb-000038
h k ∈H 2 serves as the starting position of the forged CNV.
步骤(5)根据步骤(3)和步骤(4)计算的适合伪造CNV病灶的位置、大小和轮廓，分别在该张视网膜OCT图像上及其分层金标准上进行采样和像素变换等操作，得到带CNV病灶的视网膜OCT图像及分层标签。Step (5) Based on the location, size and contour of the suitable forged CNV lesion calculated in steps (3) and (4), perform operations such as sampling and pixel transformation on the retinal OCT image and its layered gold standard respectively. Retinal OCT images with CNV lesions and hierarchical labels were obtained.
OCT图像的采样和像素变换操作详细过程参考图6，具体包括以下步骤：The detailed process of sampling and pixel transformation operations of OCT images refers to Figure 6, which specifically includes the following steps:
首先在无CNV病灶的OCT图像上定位到适合伪造CNV的位置
Figure PCTCN2022094354-appb-000039
h k∈H 2，结合适合伪造病灶的大小和轮廓，确定出伪造的CNV病灶； First, locate the location suitable for falsifying CNV on the OCT image without CNV lesions.
Figure PCTCN2022094354-appb-000039
h k ∈ H 2 , combined with the size and contour suitable for the forged lesion, determine the forged CNV lesion;
重新计算CNV病灶轮廓内各点的像素值，具体包括：CNV轮廓及轮廓内的位置集合用P[P 1,P 2...P m]表示，假设需要计算CNV轮廓内的某一点P 2(w p2,h p2)的像素值；P 1(w p1,h p1)和P 3(w p3,h p3)分别为轮廓的上某两点的位置，其中w p1＝w p3＝w p2；Q(w q,h m)是外丛装层与内核分割线M 3[W 3,H 3]上的某一点，其中w q＝w p3且w q∈H 3,h q∈H 3。采样点T(w t,h t)的像素值用Q表示，位置P 2(w p2,h p2)像素值用
Figure PCTCN2022094354-appb-000040
表示，则
Figure PCTCN2022094354-appb-000041
e为0～10之间的随机整数； Recalculate the pixel values of each point within the CNV lesion outline, including: the CNV outline and the position set within the outline are represented by P [P 1 , P 2 ...P m ]. It is assumed that a certain point P 2 within the CNV outline needs to be calculated. The pixel value of (w p2 , h p2 ); P 1 (w p1 , h p1 ) and P 3 (w p3 , h p3 ) are the positions of two points on the outline respectively, where w p1 = w p3 = w p2 ; Q(w q ,h m ) is a point on the dividing line M 3 [W 3 ,H 3 ] between the outer cladding layer and the inner core, where w q =w p3 and w q ∈H 3 ,h q ∈H 3 . The pixel value of the sampling point T (w t , h t ) is represented by Q, and the pixel value of the position P 2 (w p2 , h p2 ) is represented by
Figure PCTCN2022094354-appb-000040
means, then
Figure PCTCN2022094354-appb-000041
e is a random integer between 0 and 10;
计算出采样点T(w t,h t)的位置，进而获得采样点T(w t,h t)的像素值，并根据采样点采样点T(w t,h t)的像素值计算出点P 2(w p2,h p2)的像素值，其中，采样点T(w t,h t)的位置的计算公式为： Calculate the position of the sampling point T(w t ,h t ), and then obtain the pixel value of the sampling point T(w t ,h t ), and calculate it based on the pixel value of the sampling point T(w t ,h t ) The pixel value of point P 2 (w p2 ,h p2 ), where the calculation formula for the position of sampling point T (w t ,h t ) is:
w t＝w p2±σ,σ是(-3,3)之间的随机整数 w t =w p2 ±σ,σ is a random integer between (-3,3)
Figure PCTCN2022094354-appb-000042
Figure PCTCN2022094354-appb-000042
由于位置和像素值是对应的，当采样点T(w t,h t)的位置被计算出来后，其像素值可以基于key-value获得。 Since the position and pixel value are corresponding, when the position of the sampling point T (w t ,h t ) is calculated, its pixel value can be obtained based on key-value.
分层金标准的采样和像素变换操作和OCT图像的保持一致。The sampling and pixel transformation operations of the layered gold standard are consistent with those of OCT images.
在本发明实施例的一种具体实施方式中，生成的标签为分类标签，所述视网膜OCT图像处理模型为分类模型。在本发明实施例的其他实施方式中，生成的标签还可以是分割标签或检测标签，所述视网膜OCT图像处理模型还可以是分割模型或检测模型。In a specific implementation of the embodiment of the present invention, the generated labels are classification labels, and the retinal OCT image processing model is a classification model. In other implementations of the embodiments of the present invention, the generated labels may also be segmentation labels or detection labels, and the retinal OCT image processing model may also be a segmentation model or a detection model.
下面结合图2以及一具体实施方式对本发明实施例中的基于视网膜OCT图像的CNV病灶伪造装置进行详细说明。The CNV lesion forgery device based on retinal OCT images in the embodiment of the present invention will be described in detail below with reference to FIG. 2 and a specific embodiment.
如图2所示，基于视网膜OCT图像的CNV病灶伪造装置主要包括：CNV特征生成模块、CNV病灶伪造模块、CNV判定器和模型训练模块。CNV特征生成模块由CNV特征提取器F1、CNV特征模拟器F2以及特征扩充器A三部分组成，F1从真实的带CNV病灶的视网膜OCT图像中提取CNV病灶的轮廓、大小和位置特征，然后输入A进行仿射和弹性变换，批量扩充CNV真实特征；F2批量生成CNV模拟特征。在少量的真实的带CNV病灶的视网膜OCT图像上，使用元学习的方式训练CNV判别器C。CNV病灶伪造模块G使用CNV特征生成模块批量输出的CNV真实特征和CNV模拟特征，在大量无CNV病灶的OCT图像上伪造CNV病灶，批量生成带CNV病灶的OCT生成样本，然后送入CNV判别器判断是否合格，合格的样本会和带CNV病灶的OCT真实样本一同送入模型训练模块进行训练，不合格的生成样本会被销毁并由CNV病灶伪造模块G重新生成。As shown in Figure 2, the CNV lesion forgery device based on retinal OCT images mainly includes: CNV feature generation module, CNV lesion forgery module, CNV determiner and model training module. The CNV feature generation module consists of three parts: CNV feature extractor F1, CNV feature simulator F2 and feature expander A. F1 extracts the contour, size and location features of CNV lesions from real retinal OCT images with CNV lesions, and then inputs A performs affine and elastic transformation to expand real CNV features in batches; F2 generates CNV simulated features in batches. On a small number of real retinal OCT images with CNV lesions, the CNV discriminator C is trained using meta-learning. The CNV lesion forgery module G uses the CNV real features and CNV simulated features output in batches by the CNV feature generation module to forge CNV lesions on a large number of OCT images without CNV lesions, batch generate OCT samples with CNV lesions, and then send them to the CNV discriminator To determine whether it is qualified, qualified samples will be sent to the model training module for training together with real OCT samples with CNV lesions. Unqualified generated samples will be destroyed and regenerated by the CNV lesion forgery module G.
表1是模型训练及验证结果。真实样本是来自新一代全自动人工智能OCT BV1000拍摄的176张带CNV病灶的视网膜OCT图像。将数据划分为训练集108张，验证集34张，测试集34张。生成样本分别使用50张、100张和200张进行测试。在U-NET网络上做分割任务，不加生成样本的情况下，模型的Dice系数大约0.48±0.059，然而增加了50张、100张和200张生成样本，其Dice系数分别约为0.50±0.065、0.52±0.032，0.56±0.026增加了大约0.02、0.05、0.08。基于以上，生成样本可以有效防止模型过优化，并在一定程度上提升了模型处理效果。Table 1 shows the model training and verification results. The real samples are 176 retinal OCT images with CNV lesions taken by the new generation fully automatic artificial intelligence OCT BV1000. The data is divided into a training set of 108 images, a verification set of 34 images, and a test set of 34 images. The generated samples were tested using 50, 100 and 200 images respectively. When performing segmentation tasks on the U-NET network, without adding generated samples, the Dice coefficient of the model is approximately 0.48±0.059. However, with the addition of 50, 100 and 200 generated samples, the Dice coefficients are approximately 0.50±0.065 respectively. , 0.52±0.032, 0.56±0.026 increased by about 0.02, 0.05, 0.08. Based on the above, generating samples can effectively prevent model over-optimization and improve model processing effects to a certain extent.
表1Table 1
 ​ U-NET分割DiceU-NET Split Dice
不增加生成样本No more generated samples 0.48±0.0590.48±0.059
增加50张生成样本Add 50 generated samples 0.50±0.0650.50±0.065
增加100张生成样本Add 100 generated samples 0.53±0.0320.53±0.032
增加200张生成样本Add 200 generated samples 0.56±0.0260.56±0.026
实施例4Example 4
本发明实施例中提供了一种基于视网膜OCT图像的CNV病灶伪造系统，包括处理器及存储介质；The embodiment of the present invention provides a CNV lesion forgery system based on retinal OCT images, including a processor and a storage medium;
所述存储介质用于存储指令；The storage medium is used to store instructions;
所述处理器用于根据所述指令进行操作以执行根据实施例1任一项所述方法的步骤。The processor is configured to operate according to the instructions to perform the steps of the method according to any one of Embodiment 1.
本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。Those skilled in the art will understand that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
本申请是参照根据本申请实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.
以上结合附图对本发明的实施例进行了描述，但是本发明并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本发明的启示下，在不脱离本发明宗旨和权利要求所保护的范围情况下，还可做出很多形式，这些均属于本发明的保护之内。The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of the present invention, many forms can be made without departing from the spirit of the present invention and the scope protected by the claims, and these all fall within the protection of the present invention.
以上显示和描述了本发明的基本原理和主要特征和本发明的优点。本行业的技术人员应该了解，本发明不受上述实施例的限制，上述实施例和说明书中描述的只是说明本发明的原理，在不脱离本发明精神和范围的前提下，本发明还会有各种变化和改进，这些变化和改进都落入要求保护的本发明范围内。本发明要求保护范围由所附的权利要求书及其等效物界定。The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions only illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have other aspects. Various changes and modifications are possible, which fall within the scope of the claimed invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims (11)

  1. 一种基于视网膜OCT图像的CNV病灶伪造方法，其特征在于，包括：A method for forging CNV lesions based on retinal OCT images, which is characterized by including:
    利用真实的带CNV病灶的视网膜OCT图像，以及对应的标签，训练出CNV判别器；Use real retinal OCT images with CNV lesions and corresponding labels to train a CNV discriminator;
    基于真实的带CNV病灶的视网膜OCT图像，提取出CNV真实特征；Based on real retinal OCT images with CNV lesions, the real features of CNV are extracted;
    获取CNV模拟特征；Obtain CNV simulation characteristics;
    基于所述CNV真实特征和CNV模拟特征，在不带CNV病灶的视网膜OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器；Based on the CNV real features and CNV simulated features, forge CNV lesions on retinal OCT images without CNV lesions, batch generate retinal OCT images and labels with CNV lesions, and send them to the CNV discriminator;
    将CNV判别器输出的合格数据和真实的带CNV病灶的视网膜OCT图像送入深度学习模型进行训练，得到视网膜OCT图像处理模型。The qualified data output by the CNV discriminator and the real retinal OCT images with CNV lesions are sent to the deep learning model for training, and the retinal OCT image processing model is obtained.
  2. 根据权利要求1所述的一种基于视网膜OCT图像的CNV病灶伪造方法，其特征在于，所述真实的带CNV病灶的视网膜OCT图像数量小于第一阈值；所述CNV模拟特征的数量大于第二阈值，所述第一阈值小于第二阈值。A method for forging CNV lesions based on retinal OCT images according to claim 1, characterized in that the number of real retinal OCT images with CNV lesions is less than a first threshold; the number of CNV simulation features is greater than a second threshold Threshold, the first threshold is smaller than the second threshold.
  3. 根据权利要求1所述的一种基于视网膜OCT图像的CNV病灶伪造方法，其特征在于，所述CNV判别器的训练方法包括：A method for forging CNV lesions based on retinal OCT images according to claim 1, characterized in that the training method of the CNV discriminator includes:
    在自然数据集上使用2-ways-5query-20shot方式训练出元学习器和二分类器；Use the 2-ways-5query-20shot method to train the meta-learner and binary classifier on natural data sets;
    基于所述元学习器和二分类器，在所述真实的带CNV病灶的视网膜OCT图像上继续训练，最终得到CNV判别器。Based on the meta-learner and binary classifier, training is continued on the real retinal OCT images with CNV lesions, and finally a CNV discriminator is obtained.
  4. 根据权利要求1所述的一种基于视网膜OCT图像的CNV病灶伪造方法，其特征在于，所述CNV真实特征的提取方法包括：A CNV lesion forgery method based on retinal OCT images according to claim 1, characterized in that the extraction method of CNV real features includes:
    从所述真实的带CNV病灶的视网膜OCT图像上提取CNV病灶的大小、轮廓和位置特征；Extract the size, contour and location features of CNV lesions from the real retinal OCT images with CNV lesions;
    对CNV病灶进行仿射变换和弹性变形，并提取CNV病灶的大小、轮廓和位置特征，进行对CNV病灶的大小、轮廓和位置特征的扩充。Perform affine transformation and elastic deformation on CNV lesions, and extract the size, contour and position features of CNV lesions to expand the size, contour and position features of CNV lesions.
  5. 根据权利要求1所述的一种基于视网膜OCT图像的CNV病灶伪造方法，其特征在于：A method for forging CNV lesions based on retinal OCT images according to claim 1, characterized by:
    所述CNV模拟特征的获取方法包括：The method for obtaining the CNV simulation features includes:
    基于CNV特征模拟算法，自动生成CNV病灶的大小、轮廓和位置特征。Based on the CNV feature simulation algorithm, the size, contour and location features of CNV lesions are automatically generated.
  6. 根据权利要求1所述的一种基于视网膜OCT图像的CNV病灶伪造方法，其特征在于，所述带CNV病灶的视网膜OCT图像及标签的生成方法包括：A method for forging CNV lesions based on retinal OCT images according to claim 1, characterized in that the method for generating retinal OCT images and labels with CNV lesions includes:
    获取视网膜OCT图像及其视网膜分层金标准数据，基于所述视网膜分层金标准数据定位出视网膜各层的位置，并计算出视网膜各层的特征；Obtain the retinal OCT image and its retinal layering gold standard data, locate the positions of each retinal layer based on the retinal layering gold standard data, and calculate the characteristics of each retinal layer;
    选择一组CNV真实特征或CNV模拟特征；Select a set of CNV real features or CNV simulated features;
    若基于所述视网膜各层的特征，判断出该视网膜OCT图像适合伪造CNV病灶，则根据CNV真实特征或CNV模拟特征的位置特征，以及定位出的视网膜各层的位置，计算出适合在该张视网膜OCT图像上伪造CNV病灶的位置；If it is determined that the retinal OCT image is suitable for forging CNV lesions based on the characteristics of each retinal layer, then based on the position characteristics of the CNV real features or CNV simulated features and the located positions of each retinal layer, a calculation method is calculated that is suitable for forging CNV lesions in the image. Location of spurious CNV lesions on retinal OCT images;
    根据计算出的视网膜各层厚度，以及CNV真实特征或CNV模拟特征的大小和轮廓特征，计算出适合在该张视网膜OCT图像上伪造病灶的大小和轮廓；Based on the calculated thickness of each retinal layer, as well as the size and contour characteristics of the real CNV features or CNV simulated features, calculate the size and contour suitable for forging the lesion on the retinal OCT image;
    基于计算出的适合伪造CNV病灶的位置、大小和轮廓，分别对获取到的OCT图像及其分层金标准进行采样和像素变换，得到带CNV病灶的视网膜OCT图像及分层标签。Based on the calculated location, size and contour suitable for forged CNV lesions, the obtained OCT image and its layered gold standard were sampled and pixel transformed respectively to obtain the retinal OCT image with CNV lesions and layered labels.
  7. 根据权利要求1所述的一种基于视网膜OCT图像的CNV病灶伪造方法，其特征在于，所述对获取到的OCT图像及其分层金标准进行采样和像素变换，包括以下步骤：A method for forging CNV lesions based on retinal OCT images according to claim 1, characterized in that the sampling and pixel transformation of the acquired OCT images and their layered gold standards include the following steps:
    基于计算出的适合伪造CNV病灶的位置、大小和轮廓，确定出伪造的CNV病灶；Determine the falsified CNV lesion based on the calculated location, size, and contour of the falsified CNV lesion suitable for the falsified CNV lesion;
    分别计算伪造的CNV病灶轮廓内的各点P 2(w p2,h p2)的像素值，其中，w p2为横坐标，h p2为纵坐标；所述像素值的计算包括以下子步骤： The pixel values of each point P 2 (w p2 , h p2 ) within the forged CNV lesion outline are calculated respectively, where w p2 is the abscissa and h p2 is the ordinate; the calculation of the pixel value includes the following sub-steps:
    定义P 1(w p1,h p1)和P 3(w p3,h p3)分别为伪造的CNV病灶轮廓上某两点的位置，其中w p1和w p3分别为横坐标，h p1和h p3分别为纵坐标，w p1＝w p3＝w p2；Q(w q,h m)为视网膜外丛装层与内核分割线上的某一点，w q为该点的横坐标，h m为该点的纵坐标，其中w q＝w p3； Define P 1 (w p1 , h p1 ) and P 3 (w p3 , h p3 ) as the positions of two points on the contour of the forged CNV lesion, where w p1 and w p3 are the abscissas, h p1 and h p3 respectively. are the ordinates respectively, w p1 = w p3 = w p2 ; Q (w q , h m ) is a point on the dividing line between the outer plexus layer and the inner core of the retina, w q is the abscissa of the point, and h m is The ordinate of the point, where w q =w p3 ;
    采样点T(w t,h t)的像素值用Q表示，点P 2(w p2,h p2)像素值用
    Figure PCTCN2022094354-appb-100001
    表示，则
    Figure PCTCN2022094354-appb-100002
    e为0～10之间的随机整数；     The pixel value of sampling point T (w t , h t ) is represented by Q, and the pixel value of point P 2 (w p2 , h p2 ) is represented by
    Figure PCTCN2022094354-appb-100001
    means, then
    Figure PCTCN2022094354-appb-100002
    e is a random integer between 0 and 10;
    计算出采样点T(w t,h t)的位置，进而获得采样点T(w t,h t)的像素值，并根据采样点T(w t,h t)的像素值计算出点P 2(w p2,h p2)的像素值，其中，采样点T(w t,h t)的位置的计算公式为： Calculate the position of the sampling point T(w t ,h t ), then obtain the pixel value of the sampling point T(w t ,h t ), and calculate the point P based on the pixel value of the sampling point T(w t ,h t ) 2 (w p2 ,h p2 ) pixel value, where the calculation formula for the position of sampling point T (w t ,h t ) is:
    w t＝w p2±σ wt ＝ wp2 ±σ
    Figure PCTCN2022094354-appb-100003
    Figure PCTCN2022094354-appb-100003
    其中，σ是随机整数。where σ is a random integer.
  8. 根据权利要求1所述的一种基于视网膜OCT图像的CNV病灶伪造方法，其特征在于，生成的标签为分类标签、分割标签或检测标签，所述标签基于分层标签转换而成；所述视网膜OCT图像处理模型为分类模型、分割模型或检测模型。A method for forging CNV lesions based on retinal OCT images according to claim 1, characterized in that the generated labels are classification labels, segmentation labels or detection labels, and the labels are converted based on hierarchical labels; the retina OCT image processing models are classification models, segmentation models or detection models.
  9. 一种基于视网膜OCT图像的CNV病灶伪造装置，其特征在于，包括：A device for forging CNV lesions based on retinal OCT images, which is characterized by including:
    训练模块，用于利用所述真实的带CNV病灶的视网膜OCT图像，以及对应的标签， 训练出CNV判别器；A training module used to train a CNV discriminator using the real retinal OCT images with CNV lesions and corresponding labels;
    提取模块，用于基于所述真实的带CNV病灶的视网膜OCT图像，提取出CNV真实特征；An extraction module, configured to extract real features of CNV based on the real retinal OCT image with CNV lesions;
    获取模块，用于获取CNV模拟特征；Acquisition module, used to obtain CNV simulation features;
    伪造模块，用于基于所述CNV真实特征和CNV模拟特征，在不带CNV病灶的视网膜OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器；A forgery module, used to forge CNV lesions on retinal OCT images without CNV lesions based on the CNV real features and CNV simulated features, batch generate retinal OCT images and labels with CNV lesions, and send them to the CNV discriminator ;
    模型训练模块，用于将CNV判别器输出的合格数据和真实的带CNV病灶的视网膜OCT图像送入处理OCT图像的深度学习模型进行训练，得到视网膜OCT图像处理模型。The model training module is used to send qualified data output by the CNV discriminator and real retinal OCT images with CNV lesions into the deep learning model that processes OCT images for training, and obtain a retinal OCT image processing model.
  10. 一种基于视网膜OCT图像的CNV病灶伪造装置，其特征在于，包括：CNV特征生成模块、CNV判别器、CNV病灶伪造模块和模型训练模块；A CNV lesion forgery device based on retinal OCT images, characterized by including: a CNV feature generation module, a CNV discriminator, a CNV lesion forgery module and a model training module;
    所述CNV特征生成模块包括CNV特征提取器、CNV特征模拟器和特征扩充器，所述CNV特征提取器从真实的带CNV病灶的视网膜OCT图像中提取CNV病灶的CNV真实特征，然后输入特征扩充器进行仿射和弹性变换，批量扩充CNV病灶的特征；所述CNV特征模拟器批量生成CNV模拟特征；The CNV feature generation module includes a CNV feature extractor, a CNV feature simulator and a feature expander. The CNV feature extractor extracts the CNV real features of CNV lesions from real retinal OCT images with CNV lesions, and then inputs the feature expansion The CNV feature simulator performs affine and elastic transformation to batch-expand the features of CNV lesions; the CNV feature simulator batch-generates CNV simulation features;
    所述CNV判别器是在真实的带CNV病灶的视网膜OCT图像上使用元学习的方式训练获得；The CNV discriminator is trained using meta-learning on real retinal OCT images with CNV lesions;
    所述CNV病灶伪造模块使用CNV特征生成模块输出的CNV真实特征和CNV模拟特征，在大量无CNV病灶的OCT图像上伪造CNV病灶，批量生成带CNV病灶的视网膜OCT图像及标签，并送入所述CNV判别器进行判断；The CNV lesion forgery module uses the CNV real features and CNV simulated features output by the CNV feature generation module to forge CNV lesions on a large number of OCT images without CNV lesions, batch generate retinal OCT images and labels with CNV lesions, and send them to the institute. The CNV discriminator is used to judge;
    所述模型训练模块接收CNV判别器输出的合格的数据和真实的带CNV病灶的视网膜OCT图像，并进行训练，得到视网膜OCT图像处理模型。The model training module receives qualified data output by the CNV discriminator and real retinal OCT images with CNV lesions, and performs training to obtain a retinal OCT image processing model.
  11. 一种基于视网膜OCT图像的CNV病灶伪造系统，其特征在于，包括处理器及存储介质；A CNV lesion forgery system based on retinal OCT images, characterized by including a processor and a storage medium;
    所述存储介质用于存储指令；The storage medium is used to store instructions;
    所述处理器用于根据所述指令进行操作以执行根据权利要求1～8任一项所述方法的步骤。The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 8.
PCT/CN2022/094354 2022-05-18 2022-05-23 Cnv focus forging method, apparatus and system based on retinal oct image WO2023221149A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210542075.7A CN114947734A (en) 2022-05-18 2022-05-18 CNV focus counterfeiting method, device and system based on retina OCT image
CN202210542075.7 2022-05-18

Publications (1)

Publication Number Publication Date
WO2023221149A1 true WO2023221149A1 (en) 2023-11-23

Family

ID=82983218

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/094354 WO2023221149A1 (en) 2022-05-18 2022-05-23 Cnv focus forging method, apparatus and system based on retinal oct image

Country Status (2)

Country Link
CN (1) CN114947734A (en)
WO (1) WO2023221149A1 (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110097559A (en) * 2019-04-29 2019-08-06 南京星程智能科技有限公司 Eye fundus image focal area mask method based on deep learning
CN111695605A (en) * 2020-05-20 2020-09-22 平安科技（深圳）有限公司 Image recognition method based on OCT image, server and storage medium
US20210012487A1 (en) * 2019-07-12 2021-01-14 Mayo Foundation For Medical Education And Research Deep Learning-Based Medical Image Quality Evaluation and Virtual Clinical Trial
US20210019878A1 (en) * 2018-03-29 2021-01-21 Nec Corporation Image processing device, image processing method, and image processing program
US20210249142A1 (en) * 2018-06-11 2021-08-12 Arterys Inc. Simulating abnormalities in medical images with generative adversarial networks
EP3982324A1 (en) * 2020-10-09 2022-04-13 Koninklijke Philips N.V. Generating a sythetic image for a blood vessel lesion classifier
CN114372985A (en) * 2021-12-17 2022-04-19 中山大学中山眼科中心 Diabetic retinopathy focus segmentation method and system adapting to multi-center image

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210019878A1 (en) * 2018-03-29 2021-01-21 Nec Corporation Image processing device, image processing method, and image processing program
US20210249142A1 (en) * 2018-06-11 2021-08-12 Arterys Inc. Simulating abnormalities in medical images with generative adversarial networks
CN110097559A (en) * 2019-04-29 2019-08-06 南京星程智能科技有限公司 Eye fundus image focal area mask method based on deep learning
US20210012487A1 (en) * 2019-07-12 2021-01-14 Mayo Foundation For Medical Education And Research Deep Learning-Based Medical Image Quality Evaluation and Virtual Clinical Trial
CN111695605A (en) * 2020-05-20 2020-09-22 平安科技（深圳）有限公司 Image recognition method based on OCT image, server and storage medium
EP3982324A1 (en) * 2020-10-09 2022-04-13 Koninklijke Philips N.V. Generating a sythetic image for a blood vessel lesion classifier
CN114372985A (en) * 2021-12-17 2022-04-19 中山大学中山眼科中心 Diabetic retinopathy focus segmentation method and system adapting to multi-center image

Also Published As

Publication number Publication date
CN114947734A (en) 2022-08-30

Similar Documents

Publication Publication Date Title
Pumarola et al. Unsupervised person image synthesis in arbitrary poses
CN108520503B (en) Face defect image restoration method based on self-encoder and generation countermeasure network
CN106920227B (en) The Segmentation Method of Retinal Blood Vessels combined based on deep learning with conventional method
US20210035689A1 (en) Modeling method and apparatus for diagnosing ophthalmic disease based on artificial intelligence, and storage medium
JP2021531098A (en) Systems and methods for determining eye condition using AI
CN111259982A (en) Premature infant retina image classification method and device based on attention mechanism
WO2020103700A1 (en) Image recognition method based on micro facial expressions, apparatus and related device
CN108009222B (en) Three-dimensional model retrieval method based on better view and deep convolutional neural network
Fan et al. Principal component analysis based cataract grading and classification
CN110263755B (en) Eye ground image recognition model training method, eye ground image recognition method and eye ground image recognition device
Karki et al. Diabetic retinopathy classification using a combination of efficientnets
CN111160431B (en) Method and device for identifying keratoconus based on multi-dimensional feature fusion
CN111340776A (en) Method and system for identifying keratoconus based on multi-dimensional feature adaptive fusion
Wang et al. Cataract detection based on ocular B-ultrasound images by collaborative monitoring deep learning
Chandaliya et al. Child face age progression and regression using self-attention multi-scale patch gan
CN113627256B (en) False video inspection method and system based on blink synchronization and binocular movement detection
WO2023221149A1 (en) Cnv focus forging method, apparatus and system based on retinal oct image
Qin et al. A novel segmentation method for optic disc and optic cup based on deformable U-net
CN112818899A (en) Face image processing method and device, computer equipment and storage medium
CN111938567A (en) Deep learning-based ophthalmologic parameter measurement method, system and equipment
CN116452571A (en) Image recognition method based on deep neural network
Schlegl et al. Data-centric AI approach to improve optic nerve head segmentation and localization in OCT en face images
Singh et al. A Deep Learning Approach to Analyze Diabetic Retinopathy Lesions using Scant Data
Chen et al. DeepLab and bias field correction based automatic cone photoreceptor cell identification with adaptive optics scanning laser ophthalmoscope images
CN109509262B (en) Intelligent enhanced modeling method and device based on artificial intelligence

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22942185

Country of ref document: EP

Kind code of ref document: A1