WO2018201632A1 - 用于识别眼底图像病变的人工神经网络及系统 - Google Patents

用于识别眼底图像病变的人工神经网络及系统 Download PDF

Info

Publication number
WO2018201632A1
WO2018201632A1 PCT/CN2017/095909 CN2017095909W WO2018201632A1 WO 2018201632 A1 WO2018201632 A1 WO 2018201632A1 CN 2017095909 W CN2017095909 W CN 2017095909W WO 2018201632 A1 WO2018201632 A1 WO 2018201632A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
fundus image
fundus
image
artificial neural
Prior art date
Application number
PCT/CN2017/095909
Other languages
English (en)
French (fr)
Inventor
王娟
夏斌
白玉婧
黎晓新
胡志钢
赵瑜
Original Assignee
深圳硅基仿生科技有限公司
深圳硅基智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳硅基仿生科技有限公司, 深圳硅基智能科技有限公司 filed Critical 深圳硅基仿生科技有限公司
Priority to US16/609,213 priority Critical patent/US11213197B2/en
Publication of WO2018201632A1 publication Critical patent/WO2018201632A1/zh
Priority to US17/455,795 priority patent/US11666210B2/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/12—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/0016—Operational features thereof
    • A61B3/0025—Operational features thereof characterised by electronic signal processing, e.g. eye models
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/14—Arrangements specially adapted for eye photography
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30041—Eye; Retina; Ophthalmic
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Definitions

  • the invention relates to the field of artificial neural networks, and in particular to an artificial neural network and system for identifying lesions of fundus images.
  • the Artificial Neural Network is a machine learning model that simulates the structure of the human brain.
  • artificial neural networks especially artificial intelligence technologies such as deep learning
  • the application of artificial neural networks in the field of medical imaging diagnosis has also received more and more attention.
  • an artificial neural network it is possible to automatically determine possible lesions based on medical images and complete automatic screening of medical images.
  • artificial neural networks such as deep learning have been extensively studied in various fields such as breast cancer pathology, lung cancer detection, and cardiovascular imaging.
  • Medical images are usually obtained by cameras, X-ray transilluminators, CT, OCT or MRI, etc., which contain a wealth of details of the body structure or organization, which can help doctors and other related diagnoses by identifying these details in medical images. .
  • the fundus image includes rich details such as vitreous, retina choroid and retina choroid. If there is a related lesion in the fundus, microangioma, hemorrhage, and hard exudation will appear in the fundus image. And other lesions.
  • diabetic retinopathy as a common fundus lesion, is one of the complications of diabetes, and has become one of the main causes of blindness in adults of working age.
  • Non-Patent Document 1 discloses a method for diagnosing diabetic retinopathy using artificial intelligence, which utilizes the well-known deep learning network structure Inception-v3 for related research and obtains good results.
  • the accuracy rate has at least achieved the effect of partially replacing the work of an ophthalmologist.
  • Patent Document 2 discloses a fundus image processing method, apparatus, and system based on deep learning.
  • a conventional convolutional neural network is used to identify and analyze an image. Specifically, it uses a resampled fundus image as an input, and includes five convolutional layers and two volumes. The 7-layer convolutional neural network of the fully connected layer is identified.
  • Non-Patent Document 1 Development And Validation Of A Deep Learning Algorithm For Detection Of Diabetic Retinopathy In Retinal Fundus Photographs, JAMA November 29, 2016.
  • Patent Document 2 Chinese Patent Application Publication No. CN106408564A.
  • the Inception-v3 deep learning network structure used is a network structure for natural image classification and target detection, and requires an input image size of 299 ⁇ 299 pixels, which is not specific to Medical image.
  • Patent Document 2 relates to processing a fundus image, the purpose thereof is only to identify an area image feature, instead of making a diagnosis of a fundus disease, therefore, the fundus image processing method used in Patent Document 2 is still quite equivalent from a clinical level. distance.
  • the present invention has been made in view of the above-mentioned deficiencies of the prior art, and an object thereof is to provide an artificial neural network and system for identifying a lesion of a fundus image which can improve the accuracy of fundus image lesion determination.
  • an aspect of the present invention provides an artificial neural network for identifying a lesion of a fundus image, comprising: a preprocessing module for targeting a target eye from the same person The bottom image and the reference fundus image are respectively preprocessed; a first neural network for generating a first high-level feature set from the target fundus image; and a second neural network for generating a second advanced image from the reference fundus image a feature set module, configured to fuse the first high-level feature set with the second high-level feature set to form a feature combination set; and a third neural network, configured to generate according to the feature combination set The result of the judgment of the lesion.
  • the doctor's diagnosis process can be simulated, and the target image can be judged with reference to other fundus images from the same person, thereby facilitating Improve the accuracy of the judgment of the fundus image lesions.
  • the target fundus image and the reference fundus image may be the same. In this case, even if a fundus image is used, an effective lesion judgment result can be obtained.
  • the first neural network and the second neural network may be the same. In this case, it is possible to control the number of parameters of the neural network, improve the training efficiency of the neural network, and facilitate suppression of overfitting.
  • the pre-processing module includes: an area detecting unit for detecting a prescribed fundus area of the target fundus image and the reference fundus image; An adjustment unit for cropping and resizing the target fundus image and the reference fundus image; and a normalization unit normalizing the target fundus image and the reference fundus image.
  • the third neural network generates a judgment result of the lesion according to the feature combination set and the patient information.
  • the third neural network may comprise a fully connected layer and the patient information is an input to the fully connected layer.
  • the patient information includes at least one of age, sex, vision, and past medical history. Additionally, the patient information may also include weight. In this case, the doctor can be further simulated The diagnosis process of the birth improves the accuracy of the judgment of the lesion.
  • the first neural network and the second neural network are convolutional neural networks.
  • the convolutional neural network since the convolutional neural network has the advantages of weight sharing and local receptive field, the training of parameters can be greatly reduced, the processing speed can be improved, and the hardware overhead can be saved.
  • an artificial neural network for identifying a medical image lesion, comprising: a preprocessing module for separately preprocessing a target medical image and a reference medical image from the same person a first neural network for generating a first high-level feature set from the target medical image; a second neural network for generating a second high-level feature set from the reference medical image; a feature combining module for Combining the first high-level feature set with the second high-level feature set to form a feature combination set; and a third neural network for generating a determination result of the lesion from the feature sequence.
  • the diagnosis process of the doctor can be simulated, and the target image can be judged with reference to other medical images from the same one, thereby It is beneficial to improve the accuracy of judgment on medical image lesions.
  • the target medical image is the same as the reference medical image.
  • the neural network can be effectively trained and the judgment effect on the lesion can be improved.
  • an artificial neural network system including: a plurality of the above-described artificial neural networks; and a determiner that synthesizes and outputs the results respectively outputted from the plurality of artificial neural networks critical result.
  • another aspect of the present invention provides a method for identifying a fundus image lesion, comprising: separately performing a fundus image pair including a target fundus image and a reference fundus image; and using deep learning to identify the target a fundus image and the reference fundus image to acquire features of the target fundus image and features of the reference fundus image; combining features of the target fundus image and features of the reference fundus image to form a feature sequence; utilizing The deep learning identifies the feature sequence to obtain a judgment result of the fundus image lesion.
  • the pre-processing includes region recognition, image cropping, resizing, and normalization processing.
  • the method for identifying a fundus image lesion optionally, the pre-processing further comprises performing data amplification on the fundus image pair during training.
  • an artificial neural network and system for identifying a lesion of a fundus image and a method for identifying a lesion of a fundus image, which can improve the accuracy of fundus lesion screening.
  • FIG. 1 is a schematic view showing a lesion state of a fundus image according to the first embodiment of the present invention, wherein FIG. 1(a) shows an example of a fundus image in a normal state, and FIG. 1(b) shows An example view of the fundus image of an abnormal fundus.
  • FIG. 2 is a view showing an example of a fundus image having a fundus lesion according to the first embodiment of the present invention, wherein FIG. 2(a) shows an example of a fundus image of diabetic retinopathy, and FIG. 2(b) An exemplary diagram of a fundus image of a hypertensive fundus lesion is shown.
  • FIG. 3 is a schematic diagram showing an artificial neural network for identifying a lesion of a fundus image according to the first embodiment of the present invention.
  • FIG. 4 is a block diagram showing a preprocessing module of an artificial neural network fundus image according to the first embodiment of the present invention.
  • Fig. 5 is a schematic view showing a modification of the preprocessing module of Fig. 4.
  • FIG. 6 is a schematic diagram showing an example of a network configuration of an artificial neural network according to the first embodiment of the present invention.
  • FIG. 7 shows a schematic diagram of an example of a convolution kernel employed in the artificial neural network of FIG. 6.
  • FIG. 8 is a block diagram of an artificial neural network system according to the first embodiment of the present invention.
  • FIG. 9 is a flowchart showing a method of identifying a fundus image lesion by the artificial neural network according to the first embodiment of the present invention.
  • FIG. 10 is a block diagram showing an artificial neural network according to a second embodiment of the present invention.
  • FIG. 11 is a view showing an example of a third neural network according to a second embodiment of the present invention.
  • FIG. 12 is a block diagram showing a preprocessing module of the artificial neural network according to the second embodiment of the present invention.
  • the invention relates to an artificial neural network and system for identifying lesions of fundus images, which can improve the accuracy of fundus lesion screening.
  • a deep neural network such as a convolutional neural network is used to process a fundus image (see Non-Patent Document 1 and Patent Document 2 above), as described above, Inception-v3 is for natural image classification.
  • the network structure of the target detection is not specific to the medical image, so there is still much room for improvement in the accuracy of the clinical screening of fundus images such as screening for diabetic retinopathy.
  • the fundus image processing method employed in the above Patent Document 2 is also quite distant from the clinical level.
  • target fundus image and the reference fundus image are referred to as “eye fundus image pairs" as independent input information, that is, for " The recognition of the fundus lesion of the target fundus image, while referring to the "reference fundus image” from the same person, so it can be accurate and comprehensive The condition of fundus lesions was evaluated.
  • target fundus image refers to a fundus image that requires diagnosis of whether or not a lesion exists or what lesion exists
  • reference fundus image refers to a fundus image that is the same person as the "target fundus image”, in the present invention
  • Fig. 1 is a schematic view showing a lesion state of a fundus image according to the present embodiment, wherein Fig. 1(a) shows an example of a fundus image in a normal state, and Fig. 1(b) shows a fundus of an abnormal fundus.
  • An example diagram of an image. 2 is a view showing an example of a fundus image having a fundus lesion according to the present embodiment, in which FIG. 2(a) shows an example of a fundus image of diabetic retinopathy, and FIG. 2(b) shows a high figure.
  • An example image of a fundus image of a fundus fundus lesion An example image of a fundus image of a fundus fundus lesion.
  • the artificial neural network and system according to the present embodiment are used to learn a fundus-free fundus image (see FIG. 1(a)) and a diseased fundus image (see FIG. 1(b)).
  • Artificial neural networks and systems have the ability to determine whether there is a fundus image of a lesion. Further, in the present embodiment, the artificial neural network and the system may further learn to determine which lesion is classified and perform classification.
  • Common fundus lesions include diabetic retinopathy (see Figure 2(a)), hypertension and arteriosclerotic fundus lesions (see Figure 2(b)), age-related macular degeneration, fundus lesions, retinal vein occlusion, fundus lesions, retinal artery Obstruction of fundus lesions, high myopia fundus lesions, and even cardiovascular diseases and other related fundus lesions.
  • the artificial neural network and system according to the present embodiment are particularly suitable for diabetic retinopathy of the fundus.
  • the artificial neural network and system according to the present embodiment can realize the categories to be classified which are judged by both disease-free and disease-free, and can also realize the categories to be classified without disease and specific lesion types. Further, the types of the artificial neural network and the system to be classified according to the present embodiment may be adjusted according to specific conditions.
  • an artificial neural network or system when such an artificial neural network or system reaches the discriminating level or accuracy of the fundus physician (including sensitivity and specificity) to the relevant diagnostic criteria, it can be used to assist or replace part of the doctor's work.
  • the artificial neural network and system according to the present embodiment can save a lot of time for doctor's fundus screening (reading time), which is advantageous for making Fundus screening can be promoted and applied to promote the development of health care, especially primary health care.
  • the artificial neural network and system according to the present invention can be easily extended to identify other medical image lesions other than the fundus image lesion, where the medical image lesion can be, for example, an X-ray photograph or ultrasound applied to the body or tissue. Images, CT images, OCT images, MRI images, fluorescent contrast images, and the like.
  • FIG. 3 is a schematic diagram showing an artificial neural network 10A for identifying a lesion of a fundus image according to the present embodiment.
  • the artificial neural network 10A according to the present embodiment can be used to identify fundus image lesions.
  • the artificial neural network 10A can use a deep learning method to recognize fundus image lesions.
  • deep learning is a kind of machine learning, which is based on the representation learning of data.
  • a more abstract high-level representation attribute category or feature is formed by combining low-level features to discover a distributed feature representation of the data.
  • the accuracy of lesion recognition can be expressed by sensitivity and specificity.
  • sensitivity and specificity there are four types: true negative, true positive, false negative, and false positive.
  • True negative means that the fundus image is normal and the screening report is normal; true positive means that there is lesion in the fundus image, and the screening report shows the lesion; false negative is the lesion in the fundus image, but the screening report is normal; false positive refers to the fundus The image is normal, but the screening report incorrectly shows the lesion.
  • sensitivity and specificity are defined as follows:
  • a sensitivity of more than 80% and a specificity of 90% have been considered to be more reasonable screening modes.
  • the artificial neural network and system according to the present embodiment have a sensitivity of 85% or more and a specificity of 90% or more.
  • the fundus image lesions may include, but are not limited to, diabetic retinopathy, age-related macular degeneration fundus lesions, retinal vein occlusion fundus lesions, and the like, and are particularly suitable for diabetic retinopathy.
  • the judgment of the fundus image lesion can be processed by rating.
  • an initial rating and a secondary rating may be employed.
  • a screening report provided by the artificial neural network 10A and its system can be used as an initial rating, and then the doctor performs a secondary rating based on the screening report. Thereby, the screening result of the lesion can be obtained more accurately and reliably.
  • the neural network structure employed in the artificial neural network 10A is not particularly limited.
  • the artificial neural network 10A of the present embodiment may use a deep neural network, for example, the first neural network 12 and the second neural network 22 may adopt a structure of a deep neural network.
  • abstract image features can be extracted for a particular medical image (eg, fundus image) to aid in the determination of the lesion.
  • the artificial neural network 10A may include a preprocessing module, a first neural network 12, a second neural network 22, a feature combining module 13, and a third neural network 14.
  • the pre-processing module may specifically include a first pre-processing module 11 and a second pre-processing module 21.
  • the pre-processing modules can be used to pre-process the target fundus image and the reference fundus image (the fundus image pair) from the same person, respectively. That is, the pre-processing module 11 can pre-process the target fundus image, and the pre-processing module 21 can pre-process the reference fundus image.
  • the pre-processing module 11 and the pre-processing module 21 may be formed in the same module, or may be independently formed as a module.
  • the target fundus image and the reference fundus image from the same person are used as the input of the diagnosis, that is, the target fundus image as the first input, and the fundus image as the second input (see image 3).
  • the diagnosis of the fundus lesion of the target image not only the target fundus image itself but also the reference fundus image is used as a diagnostic reference, which simulates the doctor's comparison and reference in the actual diagnosis. The fact that a plurality of fundus images are diagnosed can improve the accuracy of the judgment of the fundus image lesions.
  • the present inventors etc.
  • the following facts are also considered: 1) different images from the same eye (target fundus image and reference fundus image) should have the same diagnosis; 2) statistically, fundus lesions from the left and right eyes of the same person (patient) similar. Therefore, when the target fundus image is diagnosed, the use of other fundus images from the patient as an aid can improve the diagnostic accuracy.
  • two fundus images of the single eye (left eye or right eye) from the same person may be used, in which case the two images may be used. Any one of the fundus images is used as the target fundus image, and the other is used as the reference fundus image. In other examples, two fundus images from the same person that belong to both eyes may also be used. Similarly, in this case, any one of the two fundus images can be used as the target fundus image, and the other as the reference fundus image.
  • the target fundus image and the reference fundus image may be the same (ie, the first input and the second input may be the same).
  • the fundus image can be used as the target fundus image and reference, respectively.
  • the fundus image can also obtain an effective lesion judgment result.
  • any one of the four fundus images can be used as the target fundus image, and the remaining three images are used as the reference fundus image.
  • multiple fundus images may be acquired during the acquisition of the fundus image.
  • any one of the plurality of fundus images can be used as the target fundus image, and the remaining fundus image can be used as the reference fundus image.
  • an equal number of fundus images from both the left and right eyes may be used.
  • the fundus image (including the target fundus image or the reference fundus image) used in the present embodiment is not particularly limited, and may be a color image (for example, an RGB image) or a grayscale image.
  • a fundus image pair composed of a target fundus image and a reference fundus image is taken as an input (first input and second input).
  • the bottom image and the reference fundus image (the fundus image pair) are approximate or identical images, and therefore, by having the target fundus image and the reference fundus image pass through the first neural network and the second neural network (ie, the target fundus image as the first input) After the first neural network, the reference fundus image is used as the second input through the second neural network, see FIG. 3) to separately extract the features of the fundus image, thereby improving the subsequent screening capability of the artificial neural network.
  • the target fundus image and the reference fundus image may belong to the fundus images of different eyes, respectively.
  • FIG. 4 is a block diagram showing a preprocessing module of the artificial neural network 10A according to the present embodiment.
  • the pre-processing module (including the first pre-processing module 11 and the second pre-processing module 21) can be used to pre-process the target fundus image and the reference fundus image (the fundus image pair) from the same person, respectively.
  • the first pre-processing module 11 and the second pre-processing module 21 may perform pre-processing such as fundus region detection, image cropping, size adjustment, normalization, and the like on the fundus image. That is, the first pre-processing module 11 can perform fundus region detection, image cropping, resizing, normalization, and the like on the target fundus image; the second pre-processing module 21 can perform fundus region detection, image cropping, and size on the reference fundus image. Adjustment, normalization, etc.
  • first pre-processing module 11 and the second pre-processing module 21 can be configured as the same module, the following only describes the first pre-processing module 11 in detail, and the structure of the second pre-processing module 12 It may be identical to the first pre-processing module 11.
  • the first pre-processing module 11 of the artificial neural network 10A mainly includes an area detecting unit 111, an adjusting unit 112, and a normalizing unit 113.
  • the area detecting unit 111 can detect the fundus area from various types of fundus images.
  • the fundus region to be detected may be, for example, a fundus region centered on the optic disc, or a fundus region including the optic disc and centered on the macula.
  • the fundus lesion can be effectively presented regardless of the region centered on the optic disc or the region including the optic disc and centered on the macula.
  • the region detecting unit 111 can detect by, for example, a sampling threshold method, a Hough transform.
  • the adjustment unit 112 can be used to trim and resize the fundus image (target fundus image). Due to the difference in the size of the human eye and the use of the fundus camera device used, the obtained fundus image may have a difference in resolution, size of the fundus region, and the like. Therefore, it is necessary to adjust these fundus images.
  • the fundus image can be tailored to a specific specification, and in some examples, a fundus fundus image such as a square can be obtained by clipping.
  • the fundus image of the present embodiment is not limited to a square shape, and may be, for example, a rectangle, a circle, an ellipse or the like.
  • the adjustment unit 112 may perform other processing for the fundus image, for example, distinguishing the fundus area on the fundus image from the patient information area (for example, some fundus images may include a name, a medical insurance number, etc.), and the adjustment is performed by different fundus camera devices using different algorithms. After the fundus image, the background of the fundus is consistent.
  • the size of the fundus image can be adjusted to a prescribed size (eg, pixel size) by the adjustment unit 112, for example, 256 ⁇ 256, 512 ⁇ 512, 1024 ⁇ 1024, and the like.
  • a prescribed size eg, pixel size
  • the present embodiment is not limited thereto, and the size of the fundus image may be any other size (pixel size) such as 128 ⁇ 128, 768 ⁇ 768, 2048 ⁇ 2048, or the like according to a specific need.
  • the image size of the fundus image of the present embodiment is preferably greater than or equal to 512 ⁇ 512 in view of being able to more accurately recognize more details of the fundus image.
  • the image size of the fundus image of the present embodiment is preferably greater than or equal to 512 ⁇ 512 in view of being able to more accurately recognize more details of the fundus image.
  • the normalization unit 113 can be used to normalize the fundus image (target fundus image). Due to the difference in fundus between different human races and the different fundus imaging devices or conditions, the fundus image may vary greatly, so it is necessary to normalize the image.
  • the normalization method of the normalization unit 113 is not particularly limited, and may be performed, for example, by zero mean, unit standard deviation, or the like. In addition, in some examples, it can also be normalized to the range of [0, 1]. By normalization, it is possible to overcome the differences in different fundus images and improve the performance of artificial neural networks.
  • FIG. 5 is a schematic view showing a modification of the preprocessing module 11 according to the present embodiment.
  • the first pre-processing module 11 may further have an amplification unit 110.
  • the amplification unit 110 may be disposed before the area detecting unit 111, but the embodiment is not limited thereto.
  • the amplification unit 110 can be used to perform data amplification on the fundus image during the training phase of the neural network. Through the amplifying unit 110, data amplification of the obtained fundus image (target fundus image) can be performed to expand the sample amount of the fundus image, thereby helping to overcome the over-fitting problem and improving the performance of the artificial neural network.
  • the amplification unit 110 is generally limited to amplifying data samples at a training phase of a neural network described later, and the amplification unit 110 may not be used at the test phase of the neural network.
  • sample amplification may be performed by performing various image transformations on the fundus image.
  • image transformation methods may include symmetric transformation, inverted transformation, rotational transformation, pixel translation, etc., and may also include adjustment of contrast, brightness, color, sharpness, and the like of the image.
  • the configuration and function of the first pre-processing module 11 have been described.
  • the second pre-processing module 21 may have exactly the same configuration and function as the first pre-processing module 11.
  • the reference presbyopia image as the second input can also be effectively preprocessed through the second pre-processing module 21 to satisfy the subsequent artificial neural network (second neural network and third neural network) for the reference fundus image. deal with.
  • the first pre-processing module 11 and the second pre-processing module 21 can perform effective pre-processing on the target fundus image and the reference fundus image, respectively, thereby facilitating further processing of the fundus images by subsequent neural networks. (eg feature extraction, etc.).
  • the first neural network 12 can be used to generate a first high-level feature set from the pre-processed target fundus image.
  • the second neural network 22 can be used to generate a second high-level feature set from the pre-processed reference fundus image.
  • the first neural network and the second neural network can realize an abstract description of the target fundus image and the reference fundus image by, for example, combining a plurality of low-level features (pixel-level features).
  • the advanced features only indicate that the primary features (eg, pixel-level features) of the original image after processing by the artificial neural network are not intended to accurately describe the high-level features of the feature, but generally, after neural network processing, The deeper the neural network, the higher the level and the more abstract.
  • the feature set generally means that two or more features are included, and may be sometimes referred to as a "characteristic matrix" in the present invention.
  • the feature set may also have only one feature such as an intermediate result, in which case the "feature set” may only refer to a single “feature”.
  • both the first neural network 12 and the second neural network 22 may employ a Convolutional Neural Network (CNN). Since the convolutional neural network has the advantages of local receptive field and weight sharing, the training of parameters can be greatly reduced, so the processing speed and hardware overhead can be improved. In addition, convolutional neural networks can more effectively handle image recognition.
  • CNN Convolutional Neural Network
  • FIG. 6 is a schematic diagram showing an example of a network configuration of an artificial neural network according to the first embodiment of the present invention.
  • FIG. 7 shows a schematic diagram of an example of a convolution kernel employed in the artificial neural network of FIG. 6.
  • a convolutional neural network may be used as the first neural network 12 and the second neural network 22, respectively.
  • the network structure of the first neural network 12 and the second neural network may be the neural network structure (simplified representation) shown in FIG. 6 and FIG. 7, respectively:
  • C represents a convolutional layer
  • S represents a pooling layer (sometimes referred to as a "downsampling layer").
  • other convolutional layers may use a 3x3 convolution kernel.
  • the increase of the training parameters can be greatly suppressed, and the training efficiency can be improved.
  • the pooling method may use max-pooling, mean-pooling, stochastic-pooling, and the like.
  • the feature dimension can be reduced, and the computing efficiency can be improved.
  • the neural network can extract more abstract high-level features to improve the accuracy of the judgment of fundus lesions.
  • the number of layers of the convoluted layer and the pooled layer may be correspondingly increased depending on the situation.
  • the neural network can also extract more abstract high-level features to further improve the accuracy of the judgment of fundus lesions.
  • the first neural network 12 and the second neural network 22 may be identical.
  • the network structure of the first neural network 12 and the network structure of the second neural network 22 may be identical. In this case, the number of parameters of the artificial neural network can be reduced, which is advantageous for suppressing over-fitting of the neural network.
  • the convolutional neural network structure employed by the first neural network 12 and the second neural network 22 is not limited thereto, and other convolutional neural network structures may be employed as long as the original fundus image (target fundus image and reference fundus) can be ensured.
  • Image Extract advanced features.
  • the first neural network 12 and the second neural network 22 according to the present embodiment are mainly used for feature extraction, and are not directly outputting the judgment result of the lesion.
  • the feature combination module 13 may be configured to fuse the first high-level feature set generated by the first neural network 12 with the second high-level feature set generated by the second neural network 22.
  • Feature combination set may mean “feature sequence”, “feature vector”, “set of feature values”, and the like, and the meaning thereof should be understood in the broadest manner.
  • feature combination module 13 may combine the first high level feature set and the second high level feature set into a one-dimensional feature vector (feature combination set). In addition, in other examples, the feature combination module 13 may also calculate a difference between the first high-level feature set and the second high-level feature set to obtain a feature combination set. Additionally, in other examples, feature combination module 13 may also calculate an average of the first high level feature set and the second high level feature set to obtain a feature combination set. Moreover, in other examples, the feature combination module 13 may perform a linear or non-linear transformation on the first high-level feature set and the second high-level feature set to obtain a feature combination set or the like.
  • the feature combination module 13 can fuse the features generated from the first neural network 12 with the features generated from the second neural network 22 to facilitate subsequent processing by the third neural network 14.
  • the third neural network 14 can be used to generate a determination result of the lesion based on the result of the feature fusion (feature combination set). As shown in FIG. 3, the third neural network 14 may form a determination result on the input target fundus image based on the result obtained by the feature combination module 13. That is, the third neural network 14 produces a judgment result for the lesion based on the feature combination set.
  • the output dimension of the third neural network 14 is consistent with the category to be classified (eg, the type of lesion). That is, for example, when the category to be classified is two types of disease-free and disease-free, the output dimension of the third neural network 14 may be 2; if the category to be classified is disease-free and specific diseases (for example, 5 types), the third nerve The output dimension of network 14 can be six. In addition, the output dimension of the third neural network 14 can be adjusted according to actual conditions.
  • the output of the third neural network 14 may be a value (percentage) between 0 and 1, which may be interpreted as the probability that the target fundus image is classified into a certain category (type of lesion). At this time, the sum of the outputs of the third neural network 14 is 1 (probability sum).
  • the output probability of the third neural network 14 is used to achieve a final diagnosis.
  • the probability of a certain category is the highest, it is determined that the fundus has a corresponding category of lesions. For example, in all categories to be classified, if the probability of no lesion is the highest, the target fundus image is judged to be free of lesions. If the probability of diabetic retinopathy is the highest, the target fundus image is judged to be diabetic retinopathy.
  • the network structure of the third neural network 14 is not particularly limited.
  • the third neural network 14 can be implemented using various combinations of convolutional layers, fully connected layers, and other auxiliary layers (eg, batch normalization, pooling, etc.).
  • the output layer of the third neural network 14 may use a single layer of convolutional layer, two layers of fully connected layers, and an output layer (softmax layer).
  • the output layer of the third neural network 14 may also use two layers of convolutional layers, two layers of pooled layers, three layers of fully connected layers, and output layers (eg, softmax layers).
  • the target fundus image and the reference fundus image are independently used as the input information, it is possible to facilitate the first neural network.
  • the landmark fundus image extracts advanced features, which facilitates the second neural network to extract advanced features from the reference fundus image.
  • the judgment result of the lesion is continuously obtained by the third neural network, whereby the diagnostic performance for the fundus image lesion can be remarkably improved.
  • the first neural network 12, the second neural network 22, and the third neural network 14 can be trained together to obtain an optimal neural network structure.
  • the fundus image pair of the training set including the target fundus image and the reference fundus image
  • the neural network is trained.
  • first neural network 12, the second neural network 22, and the third neural network 14 may be trained together at the same time, but the embodiment is not limited thereto, for example, by training self-encoding (auto The -encoder) network trains the first neural network 12 and the second neural network 22 first, and then trains with the third neural network 14.
  • self-encoding auto The -encoder
  • two single-eye images of the single eye from the same person may be used, or the same person may belong to both eyes.
  • Two fundus images may be used, or the same person may belong to both eyes.
  • four fundus images can be used, including two fundus images from the left eye and two fundus images from the right eye. In this case, it is more compatible with the true diagnosis of the fundus image lesion judgment.
  • the gold standard for the judgment of fundus image lesions currently being promoted internationally is to use seven fundus images with different fundus regions and a viewing angle of 30 degrees.
  • the present inventors have found in a long-term practice that, for example, a fundus image having four eyes of 45 degrees and a predetermined area can achieve a considerable lesion judgment effect.
  • the present embodiment is not limited thereto, and it is also possible to use more fundus images from the same person's eyes, and it is more preferable to use an equal number of fundus images from the left and right eyes.
  • a 50,000-200,000 fundus image from a cooperative hospital and removing patient information is selected as a training set, for example, 5000-20000 fundus images are tested.
  • the fundus image is pre-processed to, for example, a RGB color fundus image of 512 x 512 or 1024 x 1024 pixels.
  • the parameter adjustment is performed using a stochastic gradient descent method during training, thereby obtaining the final training result.
  • the trained artificial neural network 10A recognizes the fundus image in the test set, and obtains an average recognition accuracy rate of, for example, up to 90% or more.
  • the artificial neural network 10A according to the present embodiment can obtain an improved lesion determination accuracy rate in consideration of the clinical situation of the fundus.
  • FIG. 8 is a flowchart showing a method of identifying a fundus image lesion by the artificial neural network 10A according to the present embodiment.
  • a method of identifying a fundus image lesion by the artificial neural network 10A according to the present embodiment will be described in detail with reference to FIG.
  • a fundus image pair including a target fundus image and a reference fundus image is separately preprocessed (step S100) to obtain a fundus image satisfying a predetermined condition.
  • step S100 for example, area detection, image cropping, resizing, normalization processing, and the like can be performed on the fundus image.
  • step S100 data amplification of the fundus image pair (including the target fundus image and the reference fundus image) may be performed during the neural network training to improve the data sample size of the training, thereby improving the accuracy of determining the fundus lesion.
  • the target fundus image and the reference fundus image may be the same image.
  • the target fundus image and the reference fundus image may be separately operated by the depth learning method to acquire the feature of the target fundus image and the feature of the reference fundus image (step S200).
  • step S200 high-level features of the target fundus image and advanced features of the reference fundus image can be obtained by, for example, a convolutional neural network. Since the convolutional neural network is advantageous for having local receptive fields and weight sharing, and is advantageous for extracting advanced features of the fundus image, it is possible to improve computational efficiency and save hardware overhead.
  • the feature of the target fundus image and the feature of the reference fundus image may be fused to form a feature combination set (step S300).
  • a feature combination set facilitates integration of features of the target fundus image and features of the reference fundus image to facilitate subsequent classification and determination.
  • step S400 the feature learning set is identified by the deep learning method to obtain a judgment result of the fundus image lesion (step S400).
  • an average operation may be employed. Average Operator, Maximum Operator, Logistic Regression, Random Forest, Support Vector Machine (SVM), etc. are used to obtain the judgment results of fundus lesions.
  • SVM Support Vector Machine
  • FIG. 9 is a block diagram of the artificial neural network system 1 according to the first embodiment of the present invention.
  • a plurality of (k, k ⁇ 2) artificial nerves may be combined by combining an artificial neural network N1, an artificial neural network N2, an artificial neural network N3, ..., an artificial neural network Nk, and the like.
  • the network Ni (1 ⁇ i ⁇ k) and the determiner 40 constitute the artificial neural network system 1. That is, the artificial neural network system 1 may include a plurality of artificial neural networks (the artificial neural network N1, the artificial neural network N2, ..., the artificial neural network Nk described above) and the determiner 40.
  • the artificial neural network artificial neural network (artificial neural network N1, artificial neural network N2, artificial neural network N3, ..., artificial neural network Nk) may employ artificial neural network 10A.
  • the input of the artificial neural network Ni (1 ⁇ i ⁇ k) may be a different target fundus image and a reference fundus image (a fundus image pair) corresponding to the same eye of the same person.
  • the artificial neural network Ni (1 ⁇ i ⁇ N) may employ the artificial neural network 10A described above.
  • the artificial neural network Ni (1 ⁇ i ⁇ N) may employ different artificial neural networks 10A from the same fundus image pair.
  • the determiner 40 can synthesize the output results from the plurality of artificial neural networks Ni (1 ⁇ i ⁇ k) and output the final judgment result. That is, the outputs of the plurality of artificial neural networks (the artificial neural network N1, the artificial neural network N2, ..., the artificial neural network Nk described above) are connected to the determiner 40, and the determiner 40 outputs the final result of the output. critical result.
  • the determiner 40 may output whether there is a diseased judgment result. In other examples, the determiner 40 may output a determination as to whether there is a disease and further determine which type of fundus lesion belongs to if it is ill.
  • the determiner 40 may determine the result of the determination by outputting a probability. Additionally, in some examples, the method of the determinator 40 may employ various linear or non-linear classifiers such as Logistic Regression, Random Forest, Support Vector Machine (SVM), Adaboost, etc. In some examples, the determiner 40 may also employ some simple numerical operations, such as an Average Operator, a Maximum Operator, and the like.
  • FIG. 10 is a block diagram showing an artificial neural network 10B according to a second embodiment of the present invention.
  • FIG. 11 is a view showing an example of the third neural network 14 according to the second embodiment of the present invention.
  • FIG. 12 is a block diagram showing a third preprocessing module 31 of the artificial neural network 10B according to the second embodiment of the present invention.
  • the present embodiment relates to the artificial neural network 10B which is different from the artificial neural network 10A according to the first embodiment in that the artificial neural network 10B includes a third pre-processing module 31; the third neural network 14 can be combined with the patient according to the above-described feature set. Information to produce a judgment on the lesion (see Figure 10).
  • the artificial neural network 10B according to the present embodiment can also improve the accuracy (including sensitivity and specificity) of fundus lesion screening.
  • the feature combination set has been described in detail in the first embodiment, and therefore will not be described again in the present embodiment.
  • the feature combination set obtained by the feature combination module 13 is input to the third neural network 14. Further, the third neural network 14 generates a determination result of the lesion according to the feature combination set and the patient information.
  • the output dimension of the third neural network 14 is consistent with the category to be classified (eg, the type of lesion). That is, for example, when the category to be classified is two types of disease-free and disease-free, the output dimension of the third neural network 14 may be 2; if the category to be classified is disease-free and specific diseases (for example, 5 types), the third nerve The output dimension of network 14 can be six. In addition, the output dimension of the third neural network 14 can be adjusted according to actual conditions.
  • the output of the third neural network 14 may be a value (percentage) between 0 and 1, which may be interpreted as the probability that the target fundus image is classified into a certain category (type of lesion). At this time, the sum of the outputs of the third neural network 14 is 1 (probability sum).
  • the output probability of the third neural network 14 is used to achieve a final diagnosis.
  • the probability of a certain category is the highest, it is determined that the fundus has a corresponding category of lesions. For example, in all categories to be classified, if the probability of no lesion is the highest, the target fundus image is judged to be free of lesions. If the probability of diabetic retinopathy is the highest, the target fundus image is judged to be diabetic retinopathy.
  • patient information may include patient vision, age, gender And at least one of the past medical history.
  • patient information may also include a body weight or the like. According to the ophthalmology practice of the present inventors for many years, the patient's vision, age, sex, past medical history and body weight are closely related to fundus lesions, that is, factors such as vision, age, gender and past medical history of the patient. It is also an important reference factor for the diagnosis of fundus lesions.
  • the artificial neural network 10B may include a third pre-processing module 31 through which the patient information may be pre-processed.
  • the third pre-processing module 31 may include a feature normalization unit 311. For example, the value included in the patient information can be normalized to the [0, 1] interval, thereby avoiding patient information for subsequent nerves. Network processing can have adverse effects.
  • the patient information is added to the artificial neural network 10B as a third input to the third neural network 14A to improve the lesion recognition ability of the artificial neural network 10B.
  • patient information is also output as features to the third neural network 14.
  • the third neural network 14 can generate a judgment result on the lesion according to the feature combination set and the patient information.
  • the network structure of the third neural network 14 is not particularly limited.
  • the third neural network 14 can be implemented using various combinations of convolutional layers, fully connected layers, and other auxiliary layers (eg, batch normalization, pooling, etc.).
  • the output layer of the third neural network 14 may use a single layer of convolutional layer, two layers of fully connected layers, and an output layer (eg, a softmax layer).
  • the output layer of the third neural network 14 may also use two layers of convolutional layers, two layers of pooled layers, three layers of fully connected layers, and an output layer such as a softmax layer (see Figure 11).
  • the third neural network 14 may include a fully connected layer and patient information as an input to the fully connected layer.
  • the patient information can be used as an input to the fully connected layer (see FIG. 11).
  • the third neural network 14 when the third neural network 14 has fully connected layer patient information, it can be used as an input to its first fully connected layer, or as an input to any other fully connected layer.
  • the artificial neural network 10B simultaneously combines fundus picture information (feature combination information) and patient information for diagnosis, which is closer to the actual clinical diagnosis process of the doctor, thereby improving the accuracy of identifying the lesion of the fundus image.
  • the method steps involved in the present invention may be sequentially adjusted, combined, and deleted according to actual needs.
  • Units or subunits in the apparatus according to the present invention may be combined, divided, and deleted according to actual needs.
  • Read-Only Read-Only
  • Memory Read-Only Memory
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • PROM Programmable Read-Only Memory
  • EPROM Erasable Programmable Read Only Memory
  • OTPROM One-time Programmable Read-Only Memory
  • EEPROM Electronically-Erasable Programmable Read-Only Memory
  • CD-ROM Compact Disc Read-Only Memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Medical Informatics (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Ophthalmology & Optometry (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Quality & Reliability (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

一种用于识别眼底图像病变的人工神经网络系统，其包括：预处理模块，其用于对来自于同一个人的目标眼底图像和参考眼底图像分别进行预处理；第一神经网络（12），其用于从所述目标眼底图像产生第一高级特征集；第二神经网络（22），其用于从所述参考眼底图像产生第二高级特征集；特征组合模块（13），其用于将所述第一高级特征集与所述第二高级特征集进行融合而形成特征组合集；以及第三神经网络（14），其用于根据所述特征组合集产生对病变的判断结果。采用了目标眼底图像与参考眼底图像分别独立作为输入信息，因此，能够模拟医生的诊断过程，参考来自同一个人的其他眼底图像来对目标眼底图像进行判断，从而有利于提高对眼底图像病变的判断准确率。

Description

用于识别眼底图像病变的人工神经网络及系统 技术领域
本发明涉及人工神经网络领域，特别涉及一种用于识别眼底图像病变的人工神经网络及系统。
背景技术
人工神经网络(Artificial Neural Network)是一种模拟人脑结构的机器学习模型。随着人工神经网络，特别是深度学习等人工智能技术的发展，人工神经网络在医学影像诊断领域的应用也越来越得到关注。通过这样的人工神经网络，能够根据医学影像自动判断可能出现病变，完成对医学影像的自动筛查。例如，目前深度学习等人工神经网络已经在乳腺癌病理检查、肺癌检测、心血管成像等各个领域得到了广泛的研究。
医学影像通常通过照相机、X射线透射机、CT、OCT或MRI等来获得，其包含了丰富的身体结构或组织的诸多细节，通过识别医学影像中的这些细节，能够帮助医生等进行相关的诊断。以医学影像中的眼底图像为例，在眼底图像中包括丰富的玻璃体、视网膜脉络膜和视网膜脉络膜等的细节，如果眼底发生相关病变，则会在眼底图像中呈现出微血管瘤、出血、硬性渗出等病变。其中，例如糖尿病性视网膜病变作为常见的眼底病变是糖尿病的并发症之一，已成为工作年龄段的成年人致盲的主要原因之一。据估计，在我国，现有糖尿病患者9240万，其5年发病率为43％，致盲率为10％。各种研究表明，糖尿病性视网膜病变的早期诊断和治疗可以有效地减缓甚至改善患者的视力损伤。因此，对糖尿病病人进行定期的眼底疾病筛查具有重要的社会意义。然而，传统的糖尿病性视网膜病变的筛查需要专业的眼科医生依靠肉眼识别眼底图像来作出诊断，工作量大，人力成本高，不利于大规模地推广。同时，眼底筛查要求医生在短期内阅读大量眼底图片，可能会导致由疲劳而产生的诊断准确度下降。因此，由计算机通 过人工智能算法实现自动筛查(自动读片)变得极为需要。
目前，已有科研团队进行类似的研究，例如非专利文献1公开了一种利用人工智能进行糖尿病视网膜病变诊断的方法，其利用了著名的深度学习网络结构Inception-v3进行相关研究，并获得良好的准确率，至少实现了可以部分替代眼科专业医生工作的效果。
另外，专利文献2公开了一种基于深度学习的眼底图像处理方法、装置及系统。在专利文献2中，使用了传统的卷积神经网络对图像进行识别和分析，具体而言，其使用了重采样后的眼底图像作为输入，并采用了包括5个卷卷积层和2个全连接层的7层卷积神经网络进行识别。
[参考文献]
非专利文献1：Development And Validation Of A Deep Learning Algorithm For Detection Of Diabetic Retinopathy In Retinal Fundus Photographs,JAMA November 29,2016。
专利文献2：中国专利申请公开号CN106408564A。
发明内容
然而，在上述的现有技术中，尽管使用深度学习方法来自动识别眼底图像进行各种糖尿病视网膜病变，但是这些方法距离临床应用仍有一定距离。例如，在非专利文献1中所描述的方法中，所使用的Inception-v3深度学习网络结构是针对自然图像分类和目标检测的网络结构，要求输入图像的大小为299×299像素，并非针对特定的医学影像。
另外，尽管专利文献2也涉及对眼底图像进行处理，但是其目的仅是识别区域图像特征，而非对眼底疾病作出诊断，因此，专利文献2所使用的眼底图像处理方法离临床水平仍有相当距离。
本发明鉴于上述现有技术的不足，其目的在于提供了一种能够提高眼底图像病变判断准确率的用于识别眼底图像病变的人工神经网络及系统。
为此，本发明的一方面提供了一种用于识别眼底图像病变的人工神经网络，其包括：预处理模块，其用于对来自于同一个人的目标眼 底图像和参考眼底图像分别进行预处理；第一神经网络，其用于从所述目标眼底图像产生第一高级特征集；第二神经网络，其用于从所述参考眼底图像产生第二高级特征集；特征组合模块，其用于将所述第一高级特征集与所述第二高级特征集进行融合而形成特征组合集；以及第三神经网络，其用于根据所述特征组合集产生对病变的判断结果。
在本发明的一方面中，由于采用了目标眼底图像与参考眼底图像分别独立作为输入信息，因此，能够模拟医生的诊断过程，参考来自同一个人的其他眼底图像对目标图像进行判断，从而有利于提高对眼底图像病变的判断准确率。
另外，在本发明的一方面所涉及的人工神经网络中，所述目标眼底图像与参考眼底图像可以相同。在这种情况下，即使使用一幅眼底图像，也能够获得有效的病变判断结果。
另外，在本发明的一方面所涉及的人工神经网络中，所述第一神经网络与所述第二神经网络可以相同。在这种情况下，能够控制神经网络的参数数量，提高神经网络的训练效率，并且有利于抑制过拟合(overfitting)。
另外，在本发明的一方面所涉及的人工神经网络中，可选地，所述预处理模块包括：用于检测所述目标眼底图像和所述参考眼底图像的规定眼底区域的区域检测单元；用于对所述目标眼底图像和所述参考眼底图像进行剪裁和尺寸调整的调整单元；以及对所述目标眼底图像和所述参考眼底图像进行归一化的归一化单元。由此，能够对目标眼底图像和参考眼底图像进行有效的预处理，提高后续各个神经网络对图像特征的提取的准确度，从而改善对眼底图像病变的判断效果。
另外，在本发明的一方面所涉及的人工神经网络中，可选地，所述第三神经网络根据所述特征组合集和患者信息来产生对病变的判断结果。由此，能够更加接近医生实际诊断过程，从而能够提高判断的准确率。进一步地，所述第三神经网络可以包括全连接层，并且所述患者信息作为所述全连接层的输入。
另外，在本发明的一方面所涉及的人工神经网络中，可选地，所述患者信息包括年龄、性别、视力和既往病史当中的至少一种。另外，所述患者信息还可以包括体重。在这种情况下，能够进一步地模拟医 生的诊断过程，提高对病变判断的准确度。
另外，在本发明的一方面所涉及的人工神经网络中，可选地，所述第一神经网络和所述第二神经网络为卷积神经网络。在这种情况下，由于卷积神经网络兼具权值共享和局部感受野的优点，因此，能够极大地减小参数的训练，提高处理速度和节约硬件开销。
另外，本发明的另一方面提供了一种用于识别医学影像病变的人工神经网络，其包括：预处理模块，其用于对来自于同一个人的目标医学图像和参考医学图像分别进行预处理；第一神经网络，其用于从所述目标医学图像产生第一高级特征集；第二神经网络，其用于从所述参考医学图像产生第二高级特征集；特征组合模块，其用于将所述第一高级特征集与所述第二高级特征集进行融合而形成特征组合集；以及第三神经网络，其用于从所述特征序列产生对病变的判断结果。
在本发明的另一方面中，由于采用了目标医学图像与参考医学图像分别独立作为输入信息，因此，能够模拟医生的诊断过程，参考来自同个一个的其他医学图像对目标图像进行判断，从而有利于提高对医学图像病变的判断准确率。
在本发明的另一方面所涉及的人工神经网络中，可选地，所述目标医学影像与参考医学影像相同。在这种情况下，即使只获取同一个人的一幅眼底图像，也能够有效训练神经网络，并改善对病变的判断效果。
此外，本发明的又一方面提供了一种人工神经网络系统，包括：多个以上所述的人工神经网络；以及判断器，对从多个上述人工神经网络分别输出的结果进行综合并输出最终判断结果。
再者，本发明的其他方面还提供了一种用于识别眼底图像病变的方法，其包括：对包括目标眼底图像和参考眼底图像的眼底图像对分别进行预处理；利用深度学习识别所述目标眼底图像和所述参考眼底图像，以获取所述目标眼底图像的特征和所述参考眼底图像的特征；将所述目标眼底图像的特征和所述参考眼底图像的特征进行组合形成特征序列；利用深度学习识别所述特征序列，以获得所述眼底图像病变的判断结果。由此，能够模拟医生的诊断过程，参考来自同一个人的其他眼底图像对目标图像进行判断，从而有利于提高对眼底图像病 变的判断准确率。
另外，在本发明的其他方面所涉及用于识别眼底图像病变的方法中，可选地，所述预处理包括区域识别、图像剪裁、尺寸调整和归一化处理。
另外，在本发明的其他方面所涉及用于识别眼底图像病变的方法中，可选地，所述预处理还包括在训练时对所述眼底图像对进行数据扩增。
根据本发明，能够提供一种提高眼底病变筛查准确率的用于识别眼底图像病变的人工神经网络及系统、以及用于识别眼底图像病变的方法。
附图说明
图1示出了本发明的第1实施方式所涉及的眼底图像的病变状态的示意图，其中，图1(a)示出了正常状态的眼底图像的示例图，图1(b)示出了异常眼底的眼底图像的示例图。
图2示出了本发明的第1实施方式所涉及的具有眼底病变的眼底图像的示例图，其中，图2(a)示出了糖尿病视网膜病变的眼底图像的示例图，图2(b)示出了高血压眼底病变的眼底图像的示例图。
图3示出了本发明的第1实施方式所涉及的用于识别眼底图像病变的人工神经网络的示意图。
图4示出了本发明的第1实施方式所涉及的人工神经网络眼底图像的预处理模块的框图。
图5示出了图4的预处理模块的变形例的示意图。
图6示出了本发明的第1实施方式所涉及的人工神经网络的网络结构示例的示意图。
图7示出了图6中的人工神经网络中所采用的卷积核的示例的示意图。
图8是本发明的第1实施方式所涉及的人工神经网络系统的框图。
图9示出了本发明的第1实施方式所涉及的人工神经网络识别眼底图像病变的方法的流程图。
图10示出了本发明的第2实施方式所涉及的人工神经网络的框图。
图11示出了本发明的第2实施方式所涉及的第三神经网络的示例图。
图12示出了本发明的第2实施方式所涉及的人工神经网络的预处理模块的框图。
具体实施方式
以下，参考附图，详细地说明本发明的优选实施方式。在下面的说明中，对于相同的部件赋予相同的符号，省略重复的说明。另外，附图只是示意性的图，部件相互之间的尺寸的比例或者部件的形状等可以与实际的不同。
需要说明的是，本发明中的术语“包括”和“具有”以及它们的任何变形，例如所包括或所具有的一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可以包括或具有没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
另外，在本发明的下面描述中涉及的小标题等并不是为了限制本发明的内容或范围，其仅仅是作为阅读的提示作用。这样的小标题既不能理解为用于分割文章的内容，也不应将小标题下的内容仅仅限制在小标题的范围内。
本发明涉及能够提高眼底病变筛查准确率的用于识别眼底图像病变的人工神经网络及系统。尽管在现有技术中已经存在采用深度神经网络例如卷积神经网络来处理眼底图像的例子(参见上述非专利文献1和专利文献2)，然而如上面所述，Inception-v3是针对自然图像分类和目标检测的网络结构，并非针对特定医学影像，因此对于眼底图像临床筛查例如糖尿病性视网膜病变的筛查等在准确率上仍有很大改善的空间。另外，上述专利文献2所采用的眼底图像处理方法离临床水平也有相当距离。
相对而言，在本发明中，采用了目标眼底图像与参考眼底图像(以下有时也将“目标眼底图像和参考眼底图像”称为“眼底图像对”)分别独立作为输入信息，也即对于“目标眼底图像”的眼底病变的识别，同时参考了来自于同一个人的“参考眼底图像”，因此能够准确且全面 地评价眼底病变的状况。这里，“目标眼底图像”是指需要诊断是否存在病变或者存在什么病变的眼底图像；而“参考眼底图像”是指与“目标眼底图像”同样来自于同一个人的眼底图像，在本发明中同时使用目标眼底图像和参考眼底图像可以模拟医生的实际诊断过程，从而可以提高眼底病变判断的准确度。
[第1实施方式]
图1示出了本实施方式所涉及的眼底图像的病变状态的示意图，其中，图1(a)示出了正常状态的眼底图像的示例图，图1(b)示出了异常眼底的眼底图像的示例图。图2示出了本实施方式所涉及的具有眼底病变的眼底图像的示例图，其中，图2(a)示出了糖尿病视网膜病变的眼底图像的示例图，图2(b)示出了高血压眼底病变的眼底图像的示例图。
在本实施方式中，通过让本实施方式所涉及的人工神经网络及系统学习无病变的眼底图像(参见图1(a))和有病变的眼底图像(参见图1(b))，从而使人工神经网络及系统获得能够判断是否有病变的眼底图像的能力。另外，在本实施方式中，也可以让人工神经网络及系统进一步学习判断是哪种病变并进行分级。常见的眼底病变有糖尿病视网膜病变(参见图2(a))、高血压及动脉硬化性眼底病变(参见图2(b))、年龄相关性黄斑变性眼底病变、视网膜静脉阻塞眼底病变、视网膜动脉阻塞眼底病变、高度近视眼底病变，甚至心血管病等相关的眼底病变等。本实施方式所涉及的人工神经网络及系统特别适用于眼底的糖尿病视网膜病变。
如上所述，本实施方式所涉及的人工神经网络及系统可以实现无病和有病的两种判断的待分类类别，也可以实现无病和具体病变类型的待分类类别。另外，本实施方式所涉及的人工神经网络及系统的待分类类别也可以根据具体情况进行调整。
在一些示例中，当这样的人工神经网络或系统达到眼底医生的判别水平或准确率(包括敏感性和特异性)达到相关的诊断标准时，便能够用来辅助或替代医生的部分工作。本实施方式所涉及的人工神经网络及系统能够大量节约医生眼底筛查的时间(读片时间)，有利于使 眼底筛查能够得到推广和应用，从而推动医疗卫生特别是基层医疗卫生的发展。
另外，本发明所涉及的人工神经网络及系统也可以容易地推广到识别除眼底图像病变之外的其他医学影像病变，这里医学影像病变例如可以是针对身体或组织所进行的X光照片、超声图像、CT图像、OCT图像、MRI图像、荧光造影图像等。
图3示出了本实施方式所涉及的用于识别眼底图像病变的人工神经网络10A的示意图。例如，如图3所示，本实施方式所涉及的人工神经网络10A可以用于识别眼底图像病变，特别地，人工神经网络10A可以利用深度学习方法来识别眼底图像病变。
众所周知，深度学习是机器学习的一种，其基于对数据进行表征学习。在深度学习中，通过组合低层特征形成更加抽象的高层表示属性类别或特征，以发现数据的分布式特征表示。通过利用深度学习方法，能够提高病变识别的准确率。
在本实施方式中，病变识别的准确率可以通过敏感性和特异性来体现。具体而言，在筛查结果中，包括真阴性、真阳性、假阴性和假阳性四种。真阴性是指眼底图像正常，筛查报告也是正常；真阳性是指眼底图像存在病变，筛查报告显示出了病变；假阴性是眼底图像存在病变，但是筛查报告正常；假阳性是指眼底图像正常，但是筛查报告错误地显示有病变。于是，敏感性和特异性分别定义如下：
Figure PCTCN2017095909-appb-000001
Figure PCTCN2017095909-appb-000002
一般而言，敏感性和特异性越高，则认为病变识别的准确率越高。在一些筛查标准中，例如敏感性为80％以上和特异性为90％则已经认为是比较合理的筛查模式。相对而言，对于本实施方式所涉及的人工神经网络及系统，其敏感性可以达到85％以上，特异性可以达到90％以上。
在本实施方式中，眼底图像病变可以包括但不限于例如糖尿病视网膜病变、年龄相关性黄斑变性眼底病变、视网膜静脉阻塞眼底病变等，尤其适用于糖尿病视网膜病变。
另外，在本实施方式中，对于眼底图像病变的判断可以通过评级处理。在一些示例中，可以采用首次评级和二次评级。例如，由人工神经网络10A及其系统提供的筛查报告可以作为首次评级，然后，医生基于该筛查报告进行二次评级。由此，能够更加准确和可靠地获得病变的筛查结果。
在本实施方式中，人工神经网络10A中所采用的神经网络结构没有特别限制。在一些示例中，本实施方式所涉及的人工神经网络10A可以使用深度神经网络，例如第一神经网络12和第二神经网络22可以采用深度神经网络的结构。在这种情况下，可以针对特定医学影像(例如眼底图像)提取出抽象的图像特征，从而有助于对病变的判断。
如图3所示，本实施方式所涉及的人工神经网络10A可以包括预处理模块、第一神经网络12、第二神经网络22、特征组合模块13、以及第三神经网络14。这里，预处理模块可以具体包括第一预处理模块11和第二预处理模块21。
在本实施方式中，预处理模块(第一预处理模块11和第二预处理模块21)可以用于对来自于同一个人的目标眼底图像和参考眼底图像(眼底图像对)分别进行预处理。也即，预处理模块11可以对目标眼底图像进行预处理，预处理模块21可以对参考眼底图像进行预处理。另外，在本实施方式中，预处理模块11和预处理模块21可以在形成在同一模块中，也可以独立形成为模块。
如上所述，在本实施方式中，使用了来自于同一个人的目标眼底图像和参考眼底图像来作为诊断的输入，也即，目标眼底图像作为第一输入，参考眼底图像作为第二输入(参见图3)。如上所述，在本实施方式中，对于目标图像眼底病变的诊断，不仅使用了目标眼底图像本身，还使用了参考眼底图像作为诊断参考，该过程模拟了医生在实际诊断中会同时比较和参考多幅眼底图像进行诊断的实际情况，因此能够提高对眼底图像病变的判断准确度。
此外，在本实施方式所涉及的人工神经网络10A中，本发明人等 还考虑了以下事实：1)来自于相同眼睛的不同图像(目标眼底图像和参考眼底图像)应该有相同的诊断结果；2)从统计上看，来自同一个人(患者)的左右眼睛的眼底病变相似。因此，在对目标眼底图像进行诊断时，使用来自该患者的其他眼底图像作为辅助可以提高诊断准确度。
另外，在人工神经网络10A的训练或者测试过程中，在一些示例中，可以使用来自于同一个人的单眼(左眼或右眼)两幅眼底图像，在这种情况下，可以将这两幅眼底图像中的任意一幅作为目标眼底图像，另一幅作为参考眼底图像。在另外一些示例中，也可以使用来自于同一个人分别属于双眼的两幅眼底图像。同样地，在这种情况下，可以将这两幅眼底图像中的任意一幅作为目标眼底图像，另一幅作为参考眼底图像。
另外，在本实施方式所涉及的人工神经网络10A中，在一些示例中，目标眼底图像与参考眼底图像可以相同(即第一输入与第二输入可以相同)。在这种情况下，即使本实施方式所涉及的人工神经网络10A在训练或测试过程中仅使用来自患者的一幅眼底图像，此时，可以将这一幅眼底图像分别作为目标眼底图像和参考眼底图像，由此也能够获得有效的病变判断结果。
另外，在本实施方式中，还可以使用四幅眼底图像即包括两幅来自于左眼的眼底图像以及两幅来自于右眼的眼底图像。在这种情况下，可以将这四幅眼底图像中的任意一幅作为目标眼底图像，剩下的三幅作为参考眼底图像。
此外，在一些示例中，在眼底图像的采集过程中可以获取多幅眼底图像。在这种情况下，可以将这多幅眼底图像中的任意一幅作为目标眼底图像，剩下的眼底图像作为参考眼底图像。在另一些示例中，进一步地，可以使用来自于相等数量的来自于左右两眼的眼底图像。
另外，本实施方式所使用的眼底图像(包括目标眼底图像或参考眼底图像)没有特别限制，可以是彩色图像(例如RGB图像)，也可以是灰度图像。
在本实施方式中，将由目标眼底图像和参考眼底图像构成的眼底图像对作为输入(第一输入和第二输入)。在这种情况下，由于目标眼 底图像和参考眼底图像(眼底图像对)为近似或相同的图像，因此，通过让目标眼底图像和参考眼底图像各自经过第一神经网络和第二神经网络(即，目标眼底图像作为第一输入经过第一神经网络，参考眼底图像作为第二输入经过第二神经网络，参见图3)以分别提取眼底图像的特征，由此能够提高人工神经网络后续的筛查能力。
另外，在本实施方式所涉及的人工神经网络10A中，目标眼底图像与参考眼底图像可以分别属于不同眼睛的眼底图像。在这种情况下，有利于提高训练后的人工神经网络10A更加接近于真实的诊断情形。
(预处理模块)
图4示出了本实施方式所涉及的人工神经网络10A的预处理模块的框图。
如上所述，预处理模块(包括第一预处理模块11和第二预处理模块21)可以用于对来自于同一个人的目标眼底图像和参考眼底图像(眼底图像对)分别进行预处理。具体而言，第一预处理模块11和第二预处理模块21可以对眼底图像进行眼底区域检测、图像剪裁、尺寸调整、归一化等预处理。也即，第一预处理模块11可以对目标眼底图像进行眼底区域检测、图像剪裁、尺寸调整、归一化等；第二预处理模块21可以对参考眼底图像进行眼底区域检测、图像剪裁、尺寸调整、归一化等。
在本实施方式中，由于第一预处理模块11和第二预处理模块21可以设置成相同的模块，因此，以下仅针对第一预处理模块11进行详细描述，第二预处理模块12的结构可以与第一预处理模块11完全相同。
如图4所示，人工神经网络10A的第一预处理模块11主要包括区域检测单元111、调整单元112和归一化单元113。
在第一预处理模块11中，区域检测单元111可以从各类眼底图像中检测出眼底区域。在本实施方式中，所要检测的眼底区域例如可以是以视盘为中心的眼底区域，或者是包含视盘且以黄斑中心的眼底区域等。在本实施方式中，无论以视盘为中心的区域或者包含视盘且以黄斑为中心的区域等均能够有效地呈现眼底病变。在一些例子中，区域检测单元111可以通过例如采样阈值法、霍夫(Hough)变换来探测 眼底图像中的特定区域，以供后续人工神经网络的使用。例如，参考图1(a)和图1(b)所示的眼底图像等，眼底图像中最亮的圆圈便是视盘(optic disk)，最黑的那块是黄斑(macula)或视网膜中央凹(fovea)，还有从视盘引出的血管。
此外，如图4所示，调整单元112可以用于对眼底图像(目标眼底图像)进行剪裁和尺寸调整。由于人眼大小的不同和所使用的眼底相机设备的不同，所获得的眼底图像在分辨率、眼底区域尺寸等方面上均可能存在差异。因此，有必要对这些眼底图像进行调整。例如通过调整单元112，可以对眼底图像按照特定规格进行剪裁，在一些示例中，通过剪裁可以获得例如方形的眼底图像。另外，本实施方式的眼底图像也不限于方形，例如也可以为矩形、圆形、椭圆形等。另外，在调整单元112还可以针对眼底图像进行其他处理例如区分眼底图像上眼底区域与患者信息区域(例如有些眼底图像上可能包括姓名、医保号码等)，调整由不同眼底相机设备使用不同算法处理后的眼底图像，实现眼底背景一致化等问题。
另外，在一些示例中，通过调整单元112，能够将眼底图像的尺寸调整至规定的尺寸(例如像素尺寸)例如256×256、512×512、1024×1024等。然而，本实施方式不限于此，根据特定需要，眼底图像的尺寸也可以是任意其他规格的大小(像素大小)例如128×128、768×768、2048×2048等。
尽管本实施方式没有限定眼底图像的尺寸，但是出于能够更加准确地识别眼底图像的更多细节的方面考虑，本实施方式的眼底图像的图像尺寸优选大于或等于512×512。如上所述，在专利文献1中所涉及的深度学习框架Inception-v3中，使用了仅299×299的图像，但是由于许多眼底病症(例如糖尿病性视网膜早期病变的特征)在这样的像素级层面上的表现可能不明显，在这种情况下，可能会导致在后续下采样过程中(特别是大比例下采样时)丢失重要图像细节信息例如丢失低等级的眼底病变(例如1级糖尿病性视网膜病变)信息。基于此，在专利文献1中，1级糖尿病性视网膜病变被当作健康眼处理，如此可能会导致眼底图像病变诊断不充分，临床效果欠佳。相对而言，在本实施方式中，通过如上所述限定眼底图像的尺寸，从而能够有效地抑 制眼底图像细节信息的丢失，提高对眼底病变的判断准确度。
另外，在第一预处理模块11中，归一化单元113可以用于对眼底图像(目标眼底图像)进行归一化。由于不同人种间眼底的差异和眼底成像设备或条件的不同，眼底图像可能存在很大差异，因此，有必要针对图像进行归一化处理。
在本实施方式中，归一化单元113的归一化方式没有特别限定，例如可以采用零均值(zero mean)、单位标准方差(unit standard deviation)等进行。另外，在一些示例中，也可以归一化在[0,1]的范围内。通过归一化，能够可以克服不同眼底图像的差异性，提高人工神经网络的性能。
图5示出了本实施方式所涉及的预处理模块11的变形例的示意图。如图5所示，第一预处理模块11还可以具有扩增单元110。扩增单元110可以设置在区域检测单元111之前，但本实施方式不限于此。
在本实施方式中，扩增单元110可以用于在神经网络的训练阶段对眼底图像进行数据扩增。通过扩增单元110，可以对所获得的眼底图像(目标眼底图像)进行数据扩增以扩大眼底图像的样本量，从而有助于克服过拟合问题，提高人工神经网络的性能。另外，需要说明的是，扩增单元110一般限于在稍后描述的神经网络的训练阶段时对数据样本进行扩增，在神经网络的测试阶段时可以不使用扩增单元110。
另外，扩增单元110所采用的扩增方式没有特别限制，例如在一些示例中，可以通过对眼底图像进行各种图像变换来进行样本扩增。这样的图像变换方式可以包括对称变换、倒立变换、旋转变换、像素平移等，还可以包括对图像的对比度、亮度、颜色、锐度等进行调整。
以上，已经描述了第一预处理模块11的构成和功能，同样地，第二预处理模块21也可以具有与第一预处理模块11完全相同的构成和功能。在这种情况下，参考眼底图像作为第二输入经过第二预处理模块21也能够得到有效的预处理，以满足后续人工神经网络(第二神经网络和第三神经网络)对参考眼底图像的处理。
如上所述，通过第一预处理模块11和第二预处理模块21，能够分别对目标眼底图像和参考眼底图像进行有效的预处理，从而有助于后续各个神经网络对眼底图像中的进一步处理(例如特征提取等)。
(第一/第二神经网络)
在本实施方式中，第一神经网络12可以用于从经过预处理后的目标眼底图像产生第一高级特征集。同样地，第二神经网络22可以用于从经过预处理后的参考眼底图像产生第二高级特征集。其中，第一神经网络和第二神经网络可以通过例如组合多层低级特征(像素级特征)，实现了对目标眼底图像和参考眼底图像的抽象描述。这里，高级特征仅指示经过人工神经网络的处理后相对于原始图像的初级特征(例如像素级的特征)而言，并非为了精确描述特征的高级性，但一般而言，经过神经网络处理，随着神经网络越往深层次会呈现出越高层次和越抽象的趋势。另外，特征集一般是指包括了两个或两个以上的特征，在本发明中有时也可以称为“特征矩阵”。另外，在一些特殊的情况下，特征集也可以仅有1个特征例如中间结果，这时“特征集”可以仅特指单个“特征”。
另外，在本实施方式中，第一神经网络12和第二神经网络22均可以采用卷积神经网络(Convolutional Neural Network，CNN)。由于卷积神经网络具有局部感受野和权值共享等优点，能够极大地减小参数的训练，因此能够提高处理速度和节约硬件开销。另外，卷积神经网络能够更加有效的处理图像的识别。
图6示出了本发明的第1实施方式所涉及的人工神经网络的网络结构示例的示意图。图7示出了图6中的人工神经网络中所采用的卷积核的示例的示意图。
在一些示例中，可以使用卷积神经网络分别作为第一神经网络12和第二神经网络22。例如可以令第一神经网络12和第二神经网络的网络结构分别为图6和图7所示的神经网络结构(简化表示)：
-C1-S1-C2-S2-C3-S3-C4-
这里，C(包括C1、C2、C3和C4)表示卷积层，S(包括S1、S2和S3)表示池化(pooling)层(有时也称为“下采样层”)。在一些示例中，除了C1层使用5×5的卷积核外，其他卷积层均可以使用3×3的卷积核。在这种情况下，对于规定尺寸例如256×256、512×512的医学图像(眼底图像)，可以大大抑制了训练参数的增加，提高训练效率。
另外，在上述卷积神经网络中，池化(pooling)的方式可以使用最大池化(max-pooling)、平均池化(mean-pooling)、随机池化(stochastic-pooling)等。通过池化操作，一方面可以降低特征维度，提高运算效率，另外，也可以使神经网络提取更加抽象的高层特征，以提高对眼底病变的判断准确度。
另外，在在上述卷积神经网络中，也可以根据情况对应地增加卷积层和池化层的层数。在这种情况下，也可以使神经网络提取更加抽象的高层特征，以进一步提高对眼底病变的判断准确度。
另外，在本实施方式所涉及的人工神经网络10A中，第一神经网络12与第二神经网络22可以完全相同。具体而言，第一神经网络12的网络结构与第二神经网络22的网络结构可以完全相同。在这种情况下，能够减少人工神经网络的参数数目，有利于抑制神经网络的过拟合。
另外，第一神经网络12和第二神经网络22所采用的卷积神经网络结构不限于此，也可以采用其他卷积神经网络结构，只要能够确保从原始的眼底图像(目标眼底图像和参考眼底图像)提取出高级特征即可。此外，注意到，本实施方式所涉及的第一神经网络12和第二神经网络22主要用于特征提取，并非直接输出病变的判断结果。
(特征组合模块)
在本实施方式中，如图3所示，特征组合模块13可以用于将由第一神经网络12产生的第一高级特征集与由第二神经网络22产生的第二高级特征集进行融合而形成特征组合集。这里，本实施方式的“特征集”可以指“特征序列”、“特征矢量”、“特征值的集合”等，其意义应该以最广泛的方式理解。
在一些示例中，特征组合模块13可以将第一高级特征集和第二高级特征集组合成一维特征矢量(特征组合集)。另外，在另一些示例中，特征组合模块13也可以计算第一高级特征集与第二高级特征集的差异来获得特征组合集。另外，在另一些示例中，特征组合模块13还可以计算第一高级特征集和第二高级特征集的均值来获得特征组合集。此外，在另一些示例中，特征组合模块13可以对第一高级特征集和第二高级特征集进行线性或非线性变换来获得特征组合集等。
在本实施方式中，通过特征组合模块13，能够使从第一神经网络12产生的特征与从第二神经网络22产生的特征进行融合，便于后续第三神经网络14的处理。
(第三神经网络)
在本实施方式中，第三神经网络14可以用于根据特征融合的结果(特征组合集)产生对病变的判断结果。如图3所示，第三神经网络14可以基于特征组合模块13所获得的结果，对所输入的目标眼底图像形成判断结果。也即，第三神经网络14根据特征组合集产生对病变的判断结果。
在本实施方式中，第三神经网络14的输出维度与待分类类别(例如病变类型)一致。也即，例如待分类类别为无病和有病两种类别时，第三神经网络14的输出维度可以为2；如果待分类类别为无病和具体病症(例如5种)时，第三神经网络14的输出维度可以为6。另外，第三神经网络14的输出维度可以根据实际情况调整。
在一些示例中，第三神经网络14的输出可以是0到1之间的值(百分比)，这些值可以解释为目标眼底图像被分为某个类别(病变类型)的概率。此时，第三神经网络14的输出之和为1(概率和)。
在本实施方式中，第三神经网络14的输出概率用来实现最终诊断。在一些示例中，当某个类别的概率最高时，则判断该眼底具有对应的类别病变。例如，在所有待分类类别中，如果无病变的概率最高，则该目标眼底图像被判断为无病变。如果糖尿病视网膜病变的概率最高，则该目标眼底图像被判断为糖尿病视网膜病变。
另外，第三神经网络14的网络结构没有特别限制。在一些示例中，第三神经网络14可以使用卷积层、全连接层和其他辅助层(例如批归一化层(batch normalization)、池化层(pooling)等)的各种组合来实现。例如，在一些情况下，第三神经网络14的输出层可以使用单层的卷积层、两层全连接层和输出层(softmax层)。另外，在另一些情况下，第三神经网络14的输出层也可以使用两层卷积层、两层池化层、三层全连接层和输出层(例如softmax层)。
如上所述，在本实施方式中，由于采用了目标眼底图像与参考眼底图像分别独立作为输入信息，因此，能够有利于第一神经网络从目 标眼底图像提取出高级特征，有利于第二神经网络从参考眼底图像提取出高级特征。而且，通过将从第一神经网络和第二神经网络分别获得的高级特征组合后，继续通过第三神经网络来获得对病变的判断结果，由此能够显著地提高对眼底图像病变的诊断性能。
(训练和测试)
在本实施方式中，第一神经网络12、第二神经网络22和第三神经网络14可以一起训练，以获得最优的神经网络结构。例如在使用卷积神经网络作为第一神经网络12和第二神经网络22的情况下，在训练上述神经网络时，可以使用训练集的眼底图像对(包括目标眼底图像和参考眼底图像)对卷积神经网络进行训练。
此外，上面描述了本实施方式所涉及的第一神经网络12、第二神经网络22和第三神经网络14可以同时一起训练，但本实施方式不限于此，例如也可以通过训练自编码(auto-encoder)网络的方式先训练第一神经网络12和第二神经网络22，然后再与第三神经网络14一起训练。
另外，在本实施方式中，对于眼底图像而言，在人工神经网络10A的训练或者测试过程中，可以使用来自于同一个人的单眼两幅眼底图像，也可以使用来自于同一个人分别属于双眼的两幅眼底图像。
另外，在人工神经网络10A的训练或者测试过程中，还可以使用四幅眼底图像包括两幅来自于左眼的眼底图像以及两幅来自于右眼的眼底图像。在这种情况下，能够与眼底图像病变判断的真实诊断情况更加匹配。顺便提一下，目前国际上比较推行的眼底图像病变判断的金标准是采用七幅不同眼底区域且视角30度的眼底图像。然而，本发明人等在长期的实践发现，例如采用四幅双眼45度且规定区域的眼底图像也能达到相当的病变判断效果。
此外，本实施方式不限于此，还可以使用更多幅来自于同一个人双眼的眼底图像，更加优选地使用来自于相等数量的来自于左右两眼的眼底图像。
在本实施方式所涉及的人工神经网络10A的训练过程中，选择来自合作医院且去除患者信息的例如5-20万幅眼底图像作为训练集(training set)，例如5000-20000幅眼底图像作为测试集(testing set)。 在训练或测试过程中，眼底图像经过预处理后尺寸例如统一为512×512或1024×1024像素的RGB彩色眼底图像。
在训练过程中使用随机梯度下降法进行参数调节，由此获得最终的训练结果。然后，将训练后的人工神经网络10A对测试集中的眼底图像进行识别，获得平均识别准确率例如高达90％以上。由此可见，本实施方式所涉及的人工神经网络10A能够在兼顾眼底临床情况下获得改善的病变判断准确率。
(识别病变的流程)
图8示出了本实施方式所涉及的人工神经网络10A识别眼底图像病变的方法的流程图。以下，参考图8，详细地描述本实施方式所涉及的人工神经网络10A识别眼底图像病变的方法。
在本实施方式所涉及的用于识别眼底图像病变的方法中，首先，对包括目标眼底图像和参考眼底图像的眼底图像对分别进行预处理(步骤S100)，以获得满足规定条件的眼底图像。
在步骤S100中，例如可以对眼底图像进行区域检测、图像剪裁、尺寸调整和归一化处理等。另外，在步骤S100中，还可以在神经网络训练时对眼底图像对(包括目标眼底图像和参考眼底图像)进行数据扩增，以提高训练的数据样本量，从而提高对眼底病变判断的准确率。在一些示例中，目标眼底图像与参考眼底图像可以为相同的图像。
接着，在步骤S100之后，可以利用深度学习方法对目标眼底图像和参考眼底图像分别进行操作，以获取目标眼底图像的特征和参考眼底图像的特征(步骤S200)。在步骤S200中，可以通过例如卷积神经网络来获得目标眼底图像的高级特征和参考眼底图像的高级特征。由于卷积神经网络有利于具有局部感受野和权值共享的优点，并且有利于提取眼底图像的高级特征，因此能够提高运算效率，节约硬件开销。
在步骤S200之后，可以将目标眼底图像的特征和参考眼底图像的特征进行融合而形成特征组合集(步骤S300)。如上所述，形成特征组合集有利于目标眼底图像的特征和参考眼底图像的特征的综合，以便于后续的分类和判断。
最后，再利用深度学习方法识别特征组合集，以获得对眼底图像病变的判断结果(步骤S400)。在步骤S400中，可以采用例如平均操 作器(Average Operator)、最大值操作器(Maximum Operator)、逻辑回归(Logistic Regression)、随机森林(Random Forest)、支持向量机(SVM)等来获得对眼底病变的判断结果。
(人工神经网络系统)
图9是本发明的第1实施方式所涉及的人工神经网络系统1的框图。
在本实施方式中，如图8所示，可以通过组合人工神经网络N1、人工神经网络N2、人工神经网络N3、……、人工神经网络Nk等多个(k个，k≥2)人工神经网络Ni(1≤i≤k)和判断器40构成人工神经网络系统1。也即，人工神经网络系统1可以包括多个人工神经网络(上述的人工神经网络N1、人工神经网络N2、……、人工神经网络Nk)和判断器40。上述人工神经网络(人工神经网络N1、人工神经网络N2、人工神经网络N3、……、人工神经网络Nk)可以采用人工神经网络10A。
在本实施方式中，人工神经网络Ni(1≤i≤k)的输入可以是来自于同一个人同一只眼睛所对应的不同目标眼底图像和参考眼底图像(眼底图像对)。
另外，在一些示例中，人工神经网络Ni(1≤i≤N)均可以采用上述的人工神经网络10A。具体而言，人工神经网络Ni(1≤i≤N)可以采用使用来自于相同眼底图像对的不同人工神经网络10A。
在本实施方式中，判断器40可以对从上述多个人工神经网络Ni(1≤i≤k)的输出结果进行综合并输出最终判断结果。也即，上述多个人工神经网络(上述的人工神经网络N1、人工神经网络N2、……、人工神经网络Nk)的输出与判断器40连接，判断器40通过对输出结果的综合来输出最终判断结果。
在一些示例中，判断器40可以输出是否存在有病的判断结果。在另一些示例中，判断器40可以输出是否存在有病且进一步判断如果有病则是属于哪种类型的眼底病变的判断结果。
在一些示例中，判断器40可以通过输出概率来确定判断结果。另外，在一些示例中，判断器40的方法可以采用各种线性或非线性分类器例如逻辑回归(Logistic Regression)、随机森林(Random Forest)、 支持向量机(SVM)、Adaboost等。在一些示例中，判断器40也可以采用一些简单的数值操作，例如平均操作器(Average Operator)、最大值操作器(Maximum Operator)等。
[第2实施方式]
图10示出了本发明的第2实施方式所涉及的人工神经网络10B的框图。图11示出了本发明的第2实施方式所涉及的第三神经网络14的示例图。图12示出了本发明的第2实施方式所涉及的人工神经网络10B的第三预处理模块31的框图。
本实施方式涉及人工神经网络10B与第1实施方式所涉及的人工神经网络10A的不同点在于：人工神经网络10B包括第三预处理模块31；第三神经网络14可以根据上述特征组合集和患者信息来产生对病变的判断结果(参见图10)。本实施方式所涉及的人工神经网络10B同样能够提高眼底病变筛查准确率(包括敏感性和特异性)。
关于特征组合集已经在第1实施方式中进行了详细的描述，因此在本实施方式中不再赘述。在本实施方式中，特征组合模块13所得到的特征组合集输入到第三神经网络14，进一步地，第三神经网络14根据该特征组合集以及患者信息来产生对病变的判断结果。
在本实施方式中，第三神经网络14的输出维度与待分类类别(例如病变类型)一致。也即，例如待分类类别为无病和有病两种类别时，第三神经网络14的输出维度可以为2；如果待分类类别为无病和具体病症(例如5种)时，第三神经网络14的输出维度可以为6。另外，第三神经网络14的输出维度可以根据实际情况调整。
在一些示例中，第三神经网络14的输出可以是0到1之间的值(百分比)，这些值可以解释为目标眼底图像被分为某个类别(病变类型)的概率。此时，第三神经网络14的输出之和为1(概率和)。
在本实施方式中，第三神经网络14的输出概率用来实现最终诊断。在一些示例中，当某个类别的概率最高时，则判断该眼底具有对应的类别病变。例如，在所有待分类类别中，如果无病变的概率最高，则该目标眼底图像被判断为无病变。如果糖尿病视网膜病变的概率最高，则该目标眼底图像被判断为糖尿病视网膜病变。
另外，在一些示例中，患者信息可以包括患者视力、年龄、性别 和既往病史当中的至少一种以上。另外，患者信息还可以包括体重等。根据本发明人等在多年的眼科实践中发现，患者的视力、年龄、性别、既往病史和体重等均与眼底病变有密切的关系，也即，患者的视力、年龄、性别和既往病史等因素也是眼底病变诊断的重要参考因素。
另外，人工神经网络10B可以包括第三预处理模块31，通过第三预处理模块31可以对患者信息进行预处理。第三预处理模块31可以包括特征归一化单元311，通过特征归一化单元311，例如能够将患者信息所包括的值归一化到[0,1]区间，从而避免患者信息对后续神经网络处理可能产生的不利影响。
在本实施方式中，通过在人工神经网络10B中添加患者信息作为第三输入提供给第三神经网络14A，以提高人工神经网络10B的病变识别能力。在第三神经网络14中，除了特征组合模块13所输出的特征作为第三神经网络14的输入之外，还将患者信息作为特征输出到第三神经网络14。由此，第三神经网络14能够根据特征组合集和患者信息来产生对病变的判断结果
另外，第三神经网络14的网络结构没有特别限制。在一些示例中，第三神经网络14可以使用卷积层、全连接层和其他辅助层(例如批归一化层(batch normalization)、池化层(pooling)等)的各种组合来实现。例如，在一些情况下，第三神经网络14的输出层可以使用单层的卷积层、两层全连接层和输出层(例如softmax层)。另外，在另一些情况下，第三神经网络14的输出层也可以使用两层卷积层、两层池化层、三层全连接层和输出层例如softmax层(参见图11)。
另外，在本实施方式中，第三神经网络14可以包括全连接层，并且患者信息作为全连接层的输入。具体而言，例如当第三神经网络14使用卷积层、池化层和全连接层作为神经网络结构时，患者信息可以作为全连接层的输入(参见图11)。在本实施方式中，当第三神经网络14具有全连接层患者信息既可以作为其第一个全连接层的输入，也可以作为其他任何一个全连接层的输入。在这种情况下，人工神经网络10B同时结合眼底图片信息(特征组合信息)和患者信息进行诊断，更加接近于医生的实际临床诊断过程，从而能够提高识别眼底图像病变的准确率。
需要说明的是，对于前述的各个方法示例，为了简单描述，将其表述为一系列的动作组合，但是本领域技术人员应该知悉，本发明并不受所描述的动作顺序的限制，因为依据本申请，某一些步骤可以采用其他顺序或者同时进行。
在上述实施方式或示例中，对各个实施方式或示例的描述都各有侧重，某个实施方式或示例中没有详细描述的部分，可以参见其他实施方式或示例的相关描述。
另外，本发明所涉及的方法步骤可以根据实际需要进行顺序调整、合并和删减。本发明所涉及的装置中的单元或子单元可以根据实际需要进行合并、划分和删减。
本领域普通技术人员可以理解上述实施方式中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，该程序可以存储于计算机可读存储介质中，存储介质包括只读存储器(Read-Only Memory，ROM)、随机存储器(Random Access Memory，RAM)、可编程只读存储器(Programmable Read-only Memory，PROM)、可擦除可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)、一次可编程只读存储器(One-time Programmable Read-Only Memory，OTPROM)、电子抹除式可复写只读存储器(Electrically-Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储器、磁盘存储器、磁带存储器、或者能够用于携带或存储数据的计算机可读的任何其他介质。
虽然以上结合附图和实施例对本发明进行了具体说明，但是可以理解，上述说明不以任何形式限制本发明。本领域技术人员在不偏离本发明的实质精神和范围的情况下可以根据需要对本发明进行变形和变化，这些变形和变化均落入本发明的范围内。

Claims (11)

  1. 一种用于识别眼底图像病变的人工神经网络，其特征在于：
    包括：
    预处理模块，其用于对来自于同一个人的目标眼底图像和参考眼底图像分别进行预处理；
    第一神经网络，其用于从所述目标眼底图像产生第一高级特征集；
    第二神经网络，其用于从所述参考眼底图像产生第二高级特征集；
    特征组合模块，其用于将所述第一高级特征集与所述第二高级特征集进行融合而形成特征组合集；以及
    第三神经网络，其用于根据所述特征组合集产生对病变的判断结果。
  2. 根据权利要求1所述的人工神经网络，其特征在于：
    所述目标眼底图像与所述参考眼底图像相同。
  3. 根据权利要求1所述的人工神经网络，其特征在于：
    所述目标眼底图像与所述参考眼底图像分别属于不同眼睛的眼底图像。
  4. 根据权利要求1所述的人工神经网络，其特征在于：
    所述第一神经网络与所述第二神经网络相同。
  5. 根据权利要求1所述的人工神经网络，其特征在于：
    所述预处理模块包括：用于检测所述目标眼底图像和所述参考眼底图像的规定眼底区域的区域检测单元；用于对所述目标眼底图像和所述参考眼底图像进行剪裁和尺寸调整的调整单元；以及对所述目标眼底图像和所述参考眼底图像进行归一化的归一化单元。
  6. 根据权利要求1所述的人工神经网络，其特征在于：
    所述第三神经网络根据所述特征组合集和患者信息来产生对病变的判断结果。
  7. 根据权利要求6所述的人工神经网络，其特征在于：
    所述患者信息包括年龄、性别、视力和既往病史当中的至少一种。
  8. 根据权利要求1所述的人工神经网络，其特征在于：
    所述第一神经网络和所述第二神经网络均为卷积神经网络。
  9. 一种用于识别医学影像病变的人工神经网络，其特征在于：
    包括：
    预处理模块，其用于对来自于同一个人的目标医学图像和参考医学图像分别进行预处理；
    第一神经网络，其用于从所述目标医学图像产生第一高级特征集；
    第二神经网络，其用于从所述参考医学图像产生第二高级特征集；
    特征组合模块，其用于将所述第一高级特征集与所述第二高级特征集进行融合而形成特征组合集；以及
    第三神经网络，其用于从所述特征序列产生对病变的判断结果。
  10. 根据权利要求9所述的人工神经网络，其特征在于：
    所述目标医学影像与所述参考医学影像相同。
  11. 一种人工神经网络系统，其特征在于：
    包括：
    多个权利要求1～10中的任一项所述的人工神经网络；以及
    判断器，对从多个所述人工神经网络分别输出的结果进行综合并输出最终判断结果。
PCT/CN2017/095909 2017-05-04 2017-08-04 用于识别眼底图像病变的人工神经网络及系统 WO2018201632A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US16/609,213 US11213197B2 (en) 2017-05-04 2017-08-04 Artificial neural network and system for identifying lesion in retinal fundus image
US17/455,795 US11666210B2 (en) 2017-05-04 2021-11-19 System for recognizing diabetic retinopathy

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710306096.8 2017-05-04
CN201710306096.8A CN107045720B (zh) 2017-05-04 2017-05-04 基于人工神经网络的识别眼底图像病变的处理系统

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US16/609,213 A-371-Of-International US11213197B2 (en) 2017-05-04 2017-08-04 Artificial neural network and system for identifying lesion in retinal fundus image
US17/455,795 Continuation US11666210B2 (en) 2017-05-04 2021-11-19 System for recognizing diabetic retinopathy

Publications (1)

Publication Number Publication Date
WO2018201632A1 true WO2018201632A1 (zh) 2018-11-08

Family

ID=59546993

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/095909 WO2018201632A1 (zh) 2017-05-04 2017-08-04 用于识别眼底图像病变的人工神经网络及系统

Country Status (3)

Country Link
US (2) US11213197B2 (zh)
CN (4) CN108771530B (zh)
WO (1) WO2018201632A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109686440A (zh) * 2018-12-20 2019-04-26 深圳市新产业眼科新技术有限公司 一种在线智能诊断云平台及其运行方法和可读存储介质
WO2020205790A1 (en) * 2019-03-29 2020-10-08 Google Llc Processing fundus images using machine learning models to generate blood-related predictions
WO2021020442A1 (ja) * 2019-07-31 2021-02-04 株式会社ニコン 情報処理システム、情報処理装置、情報処理方法、及びプログラム
WO2021020444A1 (ja) * 2019-07-31 2021-02-04 株式会社ニコン 情報処理システム、情報処理装置、情報処理プログラムおよび情報処理方法
CN114998353A (zh) * 2022-08-05 2022-09-02 汕头大学·香港中文大学联合汕头国际眼科中心 一种自动检测玻璃体混浊斑飘动范围的系统

Families Citing this family (82)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108771530B (zh) * 2017-05-04 2021-03-30 深圳硅基智能科技有限公司 基于深度神经网络的眼底病变筛查系统
CN107423571B (zh) 2017-05-04 2018-07-06 深圳硅基仿生科技有限公司 基于眼底图像的糖尿病视网膜病变识别系统
WO2018217280A1 (en) 2017-05-23 2018-11-29 Walmart Apollo, Llc Automated inspection system
KR102421376B1 (ko) * 2017-06-09 2022-07-15 에스케이텔레콤 주식회사 재설정 가능한 이미지 변환을 통한 비지도된 시각적 속성 전달
CN109390053B (zh) * 2017-08-02 2021-01-08 上海市第六人民医院 眼底图像处理方法、装置、计算机设备和存储介质
US11200665B2 (en) 2017-08-02 2021-12-14 Shanghai Sixth People's Hospital Fundus image processing method, computer apparatus, and storage medium
JP7224757B2 (ja) * 2017-10-13 2023-02-20 キヤノン株式会社 診断支援装置、情報処理方法、診断支援システム及びプログラム
CN110399929B (zh) * 2017-11-01 2023-04-28 腾讯科技（深圳）有限公司 眼底图像分类方法、装置以及计算机可读存储介质
CN108399619B (zh) * 2017-12-22 2021-12-24 联想(北京)有限公司 医疗诊断的系统及装置
CN108154505A (zh) * 2017-12-26 2018-06-12 四川大学 基于深度神经网络的糖尿病视网膜病变检测方法及装置
CN108021916B (zh) * 2017-12-31 2018-11-06 南京航空航天大学 基于注意力机制的深度学习糖尿病视网膜病变分类方法
CN110037651B (zh) * 2018-01-15 2022-03-25 江威 眼底图像的质量控制方法及装置
US11080850B2 (en) * 2018-01-16 2021-08-03 Electronics And Telecommunications Research Institute Glaucoma diagnosis method using fundus image and apparatus for the same
CN108230322B (zh) * 2018-01-28 2021-11-09 浙江大学 一种基于弱样本标记的眼底特征检测装置
CN108388841B (zh) * 2018-01-30 2021-04-16 浙江大学 基于多特征深度神经网络的宫颈活检区域识别方法及装置
US11119915B2 (en) * 2018-02-08 2021-09-14 Samsung Electronics Co., Ltd. Dynamic memory mapping for neural networks
CN108470359A (zh) * 2018-02-11 2018-08-31 艾视医疗科技成都有限公司 一种糖尿病性视网膜眼底图像病变检测方法
CN108542390A (zh) * 2018-03-07 2018-09-18 清华大学 基于多对比度磁共振影像的血管斑块成分识别方法
US11448632B2 (en) 2018-03-19 2022-09-20 Walmart Apollo, Llc System and method for the determination of produce shelf life
CN108614548B (zh) * 2018-04-03 2020-08-18 北京理工大学 一种基于多模态融合深度学习的智能故障诊断方法
CN108231194A (zh) * 2018-04-04 2018-06-29 苏州医云健康管理有限公司 一种疾病诊断系统
WO2019200535A1 (zh) * 2018-04-17 2019-10-24 深圳华大生命科学研究院 基于人工智能的眼科疾病诊断建模方法、装置及系统
CN108596900B (zh) * 2018-05-02 2021-11-30 武汉联合创想科技有限公司 甲状腺相关性眼病医学影像数据处理装置、方法、计算机可读存储介质及终端设备
US11461599B2 (en) * 2018-05-07 2022-10-04 Kennesaw State University Research And Service Foundation, Inc. Classification of images based on convolution neural networks
CN108665457B (zh) * 2018-05-16 2023-12-19 腾讯医疗健康(深圳)有限公司 图像识别方法、装置、存储介质及计算机设备
JP7349425B2 (ja) * 2018-06-05 2023-09-22 住友化学株式会社 診断支援システム、診断支援方法及び診断支援プログラム
WO2020018589A1 (en) * 2018-07-16 2020-01-23 Caswell Justin Wayne Decentralized infrastructure methods and systems
WO2020023762A1 (en) 2018-07-26 2020-01-30 Walmart Apollo, Llc System and method for produce detection and classification
CN109448855A (zh) * 2018-09-17 2019-03-08 大连大学 一种基于cnn和模型融合的糖尿病血糖预测方法
RU2709661C1 (ru) * 2018-09-19 2019-12-19 Общество с ограниченной ответственностью "Аби Продакшн" Обучение нейронных сетей для обработки изображений с помощью синтетических фотореалистичных содержащих знаки изображений
CN109376767B (zh) * 2018-09-20 2021-07-13 中国科学技术大学 基于深度学习的视网膜oct图像分类方法
CN109299697A (zh) * 2018-09-30 2019-02-01 泰山学院 基于水声通信调制方式识别的深度神经网络系统及方法
JP7129870B2 (ja) * 2018-10-01 2022-09-02 富士フイルム株式会社 疾患領域を判別する判別器の学習装置、方法及びプログラム、疾患領域を判別する判別器、並びに疾患領域判別装置及びプログラム
US11715059B2 (en) 2018-10-12 2023-08-01 Walmart Apollo, Llc Systems and methods for condition compliance
CN109363698B (zh) * 2018-10-16 2022-07-12 杭州依图医疗技术有限公司 一种乳腺影像征象识别的方法及装置
CN109363697B (zh) * 2018-10-16 2020-10-16 杭州依图医疗技术有限公司 一种乳腺影像病灶识别的方法及装置
CN109376777A (zh) * 2018-10-18 2019-02-22 四川木牛流马智能科技有限公司 基于深度学习的宫颈癌组织病理学图像分析方法及设备
CN109671049B (zh) * 2018-11-07 2024-03-01 哈尔滨工业大学（深圳） 一种医学图像处理方法、系统、设备、存储介质
WO2020106332A1 (en) 2018-11-20 2020-05-28 Walmart Apollo, Llc Systems and methods for assessing products
CN109658385A (zh) * 2018-11-23 2019-04-19 上海鹰瞳医疗科技有限公司 眼底图像判断方法及设备
US10963757B2 (en) 2018-12-14 2021-03-30 Industrial Technology Research Institute Neural network model fusion method and electronic device using the same
CN109411086A (zh) * 2018-12-26 2019-03-01 深圳硅基智能科技有限公司 眼底图像病变的筛查系统
CN109886143A (zh) * 2019-01-29 2019-06-14 上海鹰瞳医疗科技有限公司 多标签分类模型训练方法和设备
CN111507981B (zh) * 2019-01-31 2021-07-13 数坤(北京)网络科技股份有限公司 图像处理方法和装置、电子设备、计算机可读存储介质
CN109998599A (zh) * 2019-03-07 2019-07-12 华中科技大学 一种基于ai技术的光/声双模成像眼底疾病诊断系统
CN111724450B (zh) * 2019-03-20 2024-07-26 上海科技大学 基于深度学习的医学图像重构系统、方法、终端、及介质
US20220156926A1 (en) * 2019-03-26 2022-05-19 Panakeia Technologies Limited A method of processing an image of tissue, a system for processing an image of tissue, a method for disease diagnosis and a disease diagnosis system
EP3948773A1 (en) * 2019-03-29 2022-02-09 Ai Technologies Inc. Image-based detection of ophthalmic and systemic diseases
CN109977905B (zh) * 2019-04-04 2021-08-06 北京百度网讯科技有限公司 用于处理眼底图像的方法和装置
CN110033422B (zh) * 2019-04-10 2021-03-23 北京科技大学 一种眼底oct图像融合方法及装置
CN110210286B (zh) * 2019-04-17 2024-05-31 平安科技（深圳）有限公司 基于眼底图像的异常识别方法、装置、设备及存储介质
CN110334575B (zh) * 2019-04-29 2022-04-26 上海交通大学 眼底照片识别方法、装置、设备及存储介质
CN110327013B (zh) * 2019-05-21 2022-02-15 北京至真互联网技术有限公司 眼底图像检测方法、装置及设备和存储介质
CN110400288B (zh) * 2019-06-18 2021-08-10 中南民族大学 一种融合双眼特征的糖网病识别方法及装置
CN110400289B (zh) * 2019-06-26 2023-10-24 平安科技（深圳）有限公司 眼底图像识别方法、装置、设备和存储介质
CN110276411B (zh) * 2019-06-28 2022-11-18 腾讯科技（深圳）有限公司 图像分类方法、装置、设备、存储介质和医疗电子设备
CN110796161B (zh) * 2019-09-18 2024-09-17 平安科技（深圳）有限公司 识别模型训练、眼底特征的识别方法、装置、设备及介质
CN110598652B (zh) * 2019-09-18 2022-04-22 上海鹰瞳医疗科技有限公司 眼底数据预测方法和设备
CN110728666B (zh) * 2019-10-08 2022-08-02 中山大学附属第三医院 基于数字病理玻片进行慢性鼻窦炎的分型方法及其系统
CN110728674B (zh) * 2019-10-21 2022-04-05 清华大学 图像处理方法及装置、电子设备和计算机可读存储介质
SG10201910949PA (en) * 2019-11-21 2020-11-27 Lian Wang Artificial Intelligence Brain
CN115511861A (zh) * 2019-11-28 2022-12-23 深圳硅基智控科技有限公司 基于人工神经网络的识别方法
CN111080630B (zh) * 2019-12-20 2024-03-08 腾讯医疗健康(深圳)有限公司 眼底图像检测设备、方法、装置及存储介质
CN111096727B (zh) * 2019-12-31 2021-02-26 上海市第十人民医院 妊高症的检测方法、系统、电子设备和存储介质
CN111260635B (zh) * 2020-01-17 2023-05-05 广州慧视医疗科技有限公司 一种全自动眼底照片获取、眼病预警及个性化管理系统
KR102306279B1 (ko) * 2020-01-29 2021-09-29 자이메드 주식회사 안저 영상판독 지원방법, 안저 영상판독 지원장치 및 이를 포함하는 안저 영상판독 지원 시스템
CN111460991A (zh) * 2020-03-31 2020-07-28 科大讯飞股份有限公司 异常检测方法、相关设备及可读存储介质
TWI734449B (zh) 2020-04-21 2021-07-21 財團法人工業技術研究院 用於影像辨識的特徵標註方法及其裝置
CN111583248B (zh) * 2020-05-12 2022-12-30 上海深至信息科技有限公司 一种基于眼部超声图像的处理方法
KR102414994B1 (ko) * 2020-06-05 2022-06-30 자이메드 주식회사 안저영상을 이용한 혈관질환 예측 지원방법 및 지원장치, 이를 포함하는 시스템
CN111640498A (zh) * 2020-06-08 2020-09-08 博奥生物集团有限公司 基于眼象数据的信息处理方法及装置
CN111754486B (zh) * 2020-06-24 2023-08-15 北京百度网讯科技有限公司 图像处理方法、装置、电子设备及存储介质
WO2022005336A1 (en) * 2020-06-29 2022-01-06 Autonomous Non-Profit Organization For Higher Education «Skolkovo Institute Of Science And Technology» Noise-resilient vasculature localization method with regularized segmentation
CN113947124A (zh) * 2020-06-30 2022-01-18 中移(成都)信息通信科技有限公司 眼底彩色图像分类模型训练方法及眼底彩色图像分类方法
CN111803024B (zh) * 2020-08-27 2022-10-04 张桦 基于深度学习算法测量眼球突出度的系统
WO2022072513A1 (en) 2020-09-30 2022-04-07 Ai-Ris LLC Retinal imaging system
TWI746287B (zh) * 2020-11-26 2021-11-11 宏碁股份有限公司 影像處理系統及影像處理方法
CN112884729B (zh) * 2021-02-04 2023-08-01 北京邮电大学 基于双模态深度学习的眼底疾病辅助诊断方法和装置
CN112869706A (zh) * 2021-02-24 2021-06-01 复旦大学 一种基于深度学习的双视野眼底图像融合方法
CN113506284B (zh) * 2021-07-26 2023-05-09 电子科技大学 一种眼底图像微血管瘤检测装置、方法及存储介质
TWI796784B (zh) * 2021-09-09 2023-03-21 宏碁智醫股份有限公司 圖像分割線補償方法及裝置
CO2022011610A1 (es) * 2022-08-17 2024-02-26 Univ Simon Bolivar Método implementado por aprendizaje de máquina para la detección de vasos sanguíneos a través de clustering de píxeles

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105513077A (zh) * 2015-12-11 2016-04-20 北京大恒图像视觉有限公司 一种用于糖尿病性视网膜病变筛查的系统
CN106408564A (zh) * 2016-10-10 2017-02-15 北京新皓然软件技术有限责任公司 一种基于深度学习的眼底图像处理方法、装置及系统
CN106408562A (zh) * 2016-09-22 2017-02-15 华南理工大学 基于深度学习的眼底图像视网膜血管分割方法及系统
CN106530295A (zh) * 2016-11-07 2017-03-22 首都医科大学 一种视网膜病变的眼底图像分类方法和装置

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN202537482U (zh) * 2012-03-28 2012-11-21 福州大学 应用于糖尿病性视网膜病变筛查的眼底照相装置
WO2013184070A1 (en) * 2012-06-05 2013-12-12 Agency For Science, Technology And Research A drusen lesion image detection system
CN103870838A (zh) * 2014-03-05 2014-06-18 南京航空航天大学 糖尿病视网膜病变的眼底图像特征提取方法
CN103871061A (zh) * 2014-03-17 2014-06-18 电子科技大学 一种基于双目视觉的眼底图像处理方法
CN104933715A (zh) * 2015-06-16 2015-09-23 山东大学（威海） 一种应用于视网膜眼底图像的配准方法
US9949714B2 (en) * 2015-07-29 2018-04-24 Htc Corporation Method, electronic apparatus, and computer readable medium of constructing classifier for disease detection
US10405739B2 (en) * 2015-10-23 2019-09-10 International Business Machines Corporation Automatically detecting eye type in retinal fundus images
CN106056595B (zh) * 2015-11-30 2019-09-17 浙江德尚韵兴医疗科技有限公司 基于深度卷积神经网络自动识别甲状腺结节良恶性的辅助诊断系统
CN105701468A (zh) * 2016-01-12 2016-06-22 华南理工大学 一种基于深度学习的人脸吸引力评价方法
JP6745496B2 (ja) * 2016-08-19 2020-08-26 学校法人自治医科大学 糖尿病網膜症の病期判定支援システムおよび糖尿病網膜症の病期の判定を支援する方法
CN106355575A (zh) * 2016-08-31 2017-01-25 上海交通大学 一种基于神经网络的监督学习型眼底血管提取方法
CN106344005B (zh) * 2016-10-28 2019-04-05 张珈绮 一种可移动心电图监测系统
US10424069B2 (en) * 2017-04-07 2019-09-24 Nvidia Corporation System and method for optical flow estimation
US10610098B1 (en) * 2017-04-11 2020-04-07 VisionQuest Biomedical LLC Generalized retinal image screening system (GRIS)
EP3616120B1 (en) * 2017-04-27 2024-09-04 Retinascan Limited System and method for automated funduscopic image analysis
CN108771530B (zh) * 2017-05-04 2021-03-30 深圳硅基智能科技有限公司 基于深度神经网络的眼底病变筛查系统
CN107423571B (zh) 2017-05-04 2018-07-06 深圳硅基仿生科技有限公司 基于眼底图像的糖尿病视网膜病变识别系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105513077A (zh) * 2015-12-11 2016-04-20 北京大恒图像视觉有限公司 一种用于糖尿病性视网膜病变筛查的系统
CN106408562A (zh) * 2016-09-22 2017-02-15 华南理工大学 基于深度学习的眼底图像视网膜血管分割方法及系统
CN106408564A (zh) * 2016-10-10 2017-02-15 北京新皓然软件技术有限责任公司 一种基于深度学习的眼底图像处理方法、装置及系统
CN106530295A (zh) * 2016-11-07 2017-03-22 首都医科大学 一种视网膜病变的眼底图像分类方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
DING, PENGLI ET AL.: "Diabetic Retinal Image Classification Method Based on Deep Neural Network", JOURNAL OF COMPUTER APPLICATIONS, vol. 37, no. 3, 10 March 2017 (2017-03-10), pages 699 - 704, ISSN: 1001-9081 *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109686440A (zh) * 2018-12-20 2019-04-26 深圳市新产业眼科新技术有限公司 一种在线智能诊断云平台及其运行方法和可读存储介质
CN109686440B (zh) * 2018-12-20 2023-12-05 深圳市新产业眼科新技术有限公司 一种在线智能诊断云平台及其运行方法和可读存储介质
WO2020205790A1 (en) * 2019-03-29 2020-10-08 Google Llc Processing fundus images using machine learning models to generate blood-related predictions
US11361435B2 (en) 2019-03-29 2022-06-14 Google Llc Processing fundus images using machine learning models to generate blood-related predictions
US11823385B2 (en) 2019-03-29 2023-11-21 Google Llc Processing fundus images using machine learning models to generate blood-related predictions
WO2021020442A1 (ja) * 2019-07-31 2021-02-04 株式会社ニコン 情報処理システム、情報処理装置、情報処理方法、及びプログラム
WO2021020444A1 (ja) * 2019-07-31 2021-02-04 株式会社ニコン 情報処理システム、情報処理装置、情報処理プログラムおよび情報処理方法
JP7558946B2 (ja) 2019-07-31 2024-10-01 株式会社ニコン 情報処理システム、情報処理装置、情報処理方法、及びプログラム
JP7558948B2 (ja) 2019-07-31 2024-10-01 株式会社ニコン 情報処理システム、情報処理装置、情報処理プログラムおよび情報処理方法
CN114998353A (zh) * 2022-08-05 2022-09-02 汕头大学·香港中文大学联合汕头国际眼科中心 一种自动检测玻璃体混浊斑飘动范围的系统
CN114998353B (zh) * 2022-08-05 2022-10-25 汕头大学·香港中文大学联合汕头国际眼科中心 一种自动检测玻璃体混浊斑飘动范围的系统

Also Published As

Publication number Publication date
CN108771530B (zh) 2021-03-30
CN107358606A (zh) 2017-11-17
US11666210B2 (en) 2023-06-06
CN107358605A (zh) 2017-11-17
CN108771530A (zh) 2018-11-09
US11213197B2 (en) 2022-01-04
CN107045720B (zh) 2018-11-30
CN107045720A (zh) 2017-08-15
US20200085290A1 (en) 2020-03-19
CN107358606B (zh) 2018-07-27
US20220079430A1 (en) 2022-03-17
CN107358605B (zh) 2018-09-21

Similar Documents

Publication Publication Date Title
WO2018201632A1 (zh) 用于识别眼底图像病变的人工神经网络及系统
US12114929B2 (en) Retinopathy recognition system
US10441160B2 (en) Method and system for classifying optic nerve head
Kauppi et al. The diaretdb1 diabetic retinopathy database and evaluation protocol.
Kauppi et al. DIARETDB0: Evaluation database and methodology for diabetic retinopathy algorithms
Wang et al. Accurate disease detection quantification of iris based retinal images using random implication image classifier technique
Phridviraj et al. A bi-directional Long Short-Term Memory-based Diabetic Retinopathy detection model using retinal fundus images
CN110992364A (zh) 视网膜图像识别方法、装置、计算机设备和存储介质
Pratumgul et al. The prototype of computer-assisted for screening and identifying severity of diabetic retinopathy automatically from color fundus images for mhealth system in Thailand
Raga A smartphone based application for early detection of diabetic retinopathy using normal eye extraction
Hakeem et al. Inception V3 and CNN Approach to Classify Diabetic Retinopathy Disease
Basheer et al. Estimation of diabetic retinopathy using deep learning
Sitienei Applying Artificial Intelligence and Mobile Technologies to Enable Practical Screening for Diabetic Retinopathy
Resendiz Automated Method for Localization and Segmentation of the Optic Disc for Glaucoma Evaluation.
Anber Diagnosing Diabetic Retinopathy from Colored Fundus Images
Meriaudeau et al. Computer aided design for diabetic retinopathy
CN115120179A (zh) 基于眼底图像检测眼底异常的判读系统及判读方法
CV et al. Iterative Kernel PCA based Classification of Retinal Images for Diabetic Macular Edema
Phridviraj et al. Healthcare Analytics
Tripathi et al. Cataract Detection Using Digital Image Processing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17908304

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17908304

Country of ref document: EP

Kind code of ref document: A1