CN116994100B

CN116994100B - Model training method and device, electronic equipment and storage medium

Info

Publication number: CN116994100B
Application number: CN202311275090.0A
Authority: CN
Inventors: 夏鹏; 马彤; 胡铭; 琚烈; 王斌; 戈宗元; 张大磊
Original assignee: Shanghai Eaglevision Medical Technology Co Ltd; Beijing Airdoc Technology Co Ltd
Current assignee: Shanghai Eaglevision Medical Technology Co Ltd; Beijing Airdoc Technology Co Ltd
Priority date: 2023-09-28
Filing date: 2023-09-28
Publication date: 2023-12-22
Anticipated expiration: 2043-09-28
Also published as: CN116994100A

Abstract

The application provides a model training method, a model training device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a sample fundus image of a user, which accords with model training conditions, wherein the sample fundus image is marked with the real disease category of Alzheimer's disease; inputting a sample fundus image into a disease prediction model to be trained; invoking the feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling treatment on the sample fundus image to obtain an attention feature map and a blood vessel labeling feature map; calling a feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map to obtain a predicted disease probability corresponding to the sample fundus image; calculating a loss value of a disease prediction model to be trained based on the real disease category and the predicted disease probability; and under the condition that the loss value is in a preset range, obtaining a disease prediction model for predicting the Alzheimer disease. The method and the device can improve the prediction accuracy of the Alzheimer's disease.

Description

Model training method and device, electronic equipment and storage medium

Technical Field

The application relates to the technical field of artificial intelligence, in particular to a model training method, a model training device, electronic equipment and a storage medium.

Background

Alzheimer's Disease (Alzheimer's Disease) is a progressive degenerative brain Disease mainly represented by comprehensive cognitive decline, and comprises hypoability and personality changes of learning, memory, language, executive function, complex attention and the like, the Disease is hidden, conscious disturbance is not accompanied, most of the Disease course is irreversible, and the Alzheimer's Disease is one of serious diseases threatening the health of the old, and no effective therapeutic drug exists at present.

In recent years, many research efforts have effectively utilized deep learning techniques on fundus images, exploring the feasibility of screening and predicting some diseases. However, the existing work at present has great limitation, and the following main problems mainly exist:

1. the scarcity of Alzheimer's disease patient data;

2. obvious features associated with Alzheimer's disease in fundus images are difficult to observe, thus increasing the difficulty of model learning.

Based on the above, in the prior art, because the data of the patient with the Alzheimer's disease is scarce and the obvious features related to the Alzheimer's disease in the fundus image are difficult to observe, the training difficulty of the Alzheimer's disease prediction model is increased, and the prediction accuracy of the Alzheimer's disease prediction model obtained by training is lower.

Disclosure of Invention

The embodiment of the application provides a model training method, device, electronic equipment and storage medium, which are used for solving the problems that in the prior art, due to the fact that Alzheimer's disease patient data are scarce and obvious characteristics related to Alzheimer's disease in fundus images are difficult to observe, training difficulty of an Alzheimer's disease prediction model is increased, and prediction accuracy of the Alzheimer's disease prediction model obtained through training is low.

In order to solve the above technical problems, embodiments of the present application are implemented as follows:

in a first aspect, an embodiment of the present application provides a model training method, where the method includes:

acquiring a sample fundus image of a user, which accords with model training conditions, wherein the sample fundus image is marked with the real disease category of Alzheimer's disease;

inputting the sample fundus image to a disease prediction model to be trained, the disease prediction model to be trained comprising: feature screening, a blood vessel segmentation layer and a feature fusion layer;

invoking the feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling treatment on the sample fundus image to obtain an attention feature map and a blood vessel labeling feature map;

Invoking the feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map to obtain predicted disease probability corresponding to the sample fundus image;

calculating a loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability;

and under the condition that the loss value is in a preset range, obtaining a disease prediction model for predicting Alzheimer's disease.

Optionally, the feature screening and blood vessel segmentation layer includes: an attention layer and a blood vessel segmentation layer,

the invoking the feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling processing on the sample fundus image to obtain an attention feature map and a blood vessel labeling feature map, comprising the following steps:

calling the attention layer to perform feature screening processing on the sample fundus image, and outputting an attention feature map;

and calling the blood vessel segmentation layer to perform blood vessel segmentation labeling processing on the sample fundus image, and outputting a blood vessel labeling feature map.

Optionally, the attention layer includes: the device comprises N convolution layers, a feature map processing layer and an output layer, wherein parameters of the N convolution layers are different;

The calling the attention layer performs feature screening processing on the sample fundus image, and outputs an attention feature map, which comprises the following steps:

calling the N convolution layers to respectively carry out channel compression processing on the sample fundus image to generate N branch feature images;

calling the feature map processing layer to process the N branch feature maps to generate a fusion feature map;

and calling the output layer to process the fusion feature map and outputting the attention feature map.

Optionally, when n=3, the N branch feature graphs include: a first branch feature map, a second branch feature map, and a third branch feature map,

the calling the feature map processing layer to process the N branch feature maps to generate a fusion feature map, including:

transferring the first branch feature map by calling the feature map processing layer to obtain a first transfer feature map;

performing matrix multiplication on the first transfer feature map and the second branch feature map to obtain an intermediate feature map;

normalizing the intermediate feature map to obtain a normalized feature map;

and carrying out matrix multiplication on the normalized feature map and the third branch feature map to obtain the fusion feature map.

Optionally, the invoking the blood vessel segmentation layer performs blood vessel segmentation labeling processing on the sample fundus image, and outputs a blood vessel labeling feature map, including:

invoking the blood vessel segmentation layer to perform blood vessel segmentation labeling treatment on blood vessels in the sample fundus image to obtain a blood vessel segmentation feature map;

and carrying out pooling treatment on the blood vessel segmentation feature map, and outputting the blood vessel labeling feature map with the same dimension as the attention feature map.

Optionally, the invoking the feature fusion layer performs feature fusion processing on the attention feature map and the blood vessel labeling feature map to obtain a predicted disease probability corresponding to the sample fundus image, including:

invoking the feature fusion layer to perform matrix point multiplication processing on the attention feature map and the blood vessel labeling feature map to obtain a point multiplication feature map;

performing cross multiplication on the point multiplication feature map and the attention feature map to obtain a target feature map;

and determining the predicted disease probability corresponding to the sample fundus image based on the target feature map.

Optionally, the calculating, based on the real disease category and the predicted disease probability, a loss value of the disease prediction model to be trained includes:

Calculating a cross entropy loss value based on the real disease category and the predicted disease probability;

and calculating the loss value of the disease prediction model to be trained based on the number of samples in the round of training and the cross entropy loss value.

Optionally, the acquiring the sample fundus image of the user conforming to the model training condition includes:

acquiring a plurality of initial fundus images of the user;

and calling a preset model to process the initial fundus image so as to screen out the sample fundus image which accords with the model training condition from the initial fundus image, wherein the sample fundus image is a real fundus image type and has an image quality greater than a quality threshold.

Optionally, the invoking the preset model to process the initial fundus image to screen the sample fundus image meeting the model training condition in the initial fundus image includes:

preprocessing the initial fundus image to generate a preprocessed fundus image;

invoking a fundus image recognition model to process the preprocessed fundus image to obtain a fundus image recognition result;

determining a standard fundus image in the initial fundus image according to the fundus image identification result;

Invoking an image quality grade classification model to process the preprocessed fundus image to obtain the probability that the preprocessed fundus image belongs to each preset quality grade;

and screening sample fundus images meeting model training conditions from the standard fundus images according to the probability.

In a second aspect, embodiments of the present application provide a model training apparatus, the apparatus including:

the sample image acquisition module is used for acquiring a sample fundus image of a user, which accords with the model training condition, and the sample fundus image is marked with the real disease category of the Alzheimer disease;

the sample image input module is used for inputting the sample fundus image into a disease prediction model to be trained, and the disease prediction model to be trained comprises: feature screening, a blood vessel segmentation layer and a feature fusion layer;

the marked feature map acquisition module is used for calling the feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation marking processing on the sample fundus image to obtain an attention feature map and a blood vessel marked feature map;

the prediction type acquisition module is used for calling the feature fusion layer to perform feature fusion processing on the attention feature image and the blood vessel labeling feature image so as to obtain a prediction disease probability corresponding to the sample fundus image;

The loss value calculation module is used for calculating the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability;

the prediction model acquisition module is used for obtaining a disease prediction model for predicting the Alzheimer disease under the condition that the loss value is in a preset range.

the labeling feature diagram acquisition module comprises:

the attention characteristic map output unit is used for calling the attention layer to perform characteristic screening processing on the sample fundus image and outputting an attention characteristic map;

and the labeling feature map output unit is used for calling the blood vessel segmentation layer to perform blood vessel segmentation labeling processing on the sample fundus image and outputting a blood vessel labeling feature map.

the attention attempt output unit includes:

the branch characteristic diagram generating subunit is used for calling the N convolution layers to respectively perform channel compression processing on the sample fundus image to generate N branch characteristic diagrams;

The fusion feature map generation subunit is used for calling the feature map processing layer to process the N branch feature maps to generate a fusion feature map;

and the attention characteristic diagram output subunit is used for calling the output layer to process the fusion characteristic diagram and outputting the attention characteristic diagram.

the fusion feature map generation unit includes:

a transposed feature map acquisition subunit, configured to invoke the feature map processing layer to transpose the first branch feature map to obtain a first transposed feature map;

an intermediate feature map obtaining subunit, configured to perform matrix multiplication on the first transformation feature map and the second branch feature map to obtain an intermediate feature map;

the normalized feature map acquisition subunit is used for carrying out normalization processing on the intermediate feature map to obtain a normalized feature map;

and the fusion feature map acquisition subunit is used for carrying out matrix multiplication on the normalized feature map and the third branch feature map to obtain the fusion feature map.

Optionally, the labeling feature map output unit includes:

The segmentation feature map acquisition subunit is used for calling the blood vessel segmentation layer to perform blood vessel segmentation labeling treatment on blood vessels in the sample fundus image so as to obtain a blood vessel segmentation feature map;

and the labeling feature map output subunit is used for carrying out pooling processing on the blood vessel segmentation feature map and outputting the blood vessel labeling feature map with the same dimension as the attention feature map.

Optionally, the prediction category acquisition module includes:

the dot-product feature map acquisition unit is used for calling the feature fusion layer to perform matrix dot-product processing on the attention feature map and the blood vessel labeling feature map to obtain a dot-product feature map;

the target feature map acquisition unit is used for carrying out cross multiplication processing on the point multiplication feature map and the attention feature map to obtain a target feature map;

and the prediction type determining unit is used for determining the prediction disease probability corresponding to the sample fundus image based on the target feature map.

Optionally, the loss value calculation module includes:

the cross entropy loss calculation unit is used for calculating a cross entropy loss value based on the real disease category and the predicted disease probability;

and the loss value calculation unit is used for calculating the loss value of the disease prediction model to be trained based on the number of samples in the round of training and the cross entropy loss value.

Optionally, the sample image acquisition module includes:

an initial image acquisition unit configured to acquire a plurality of initial fundus images of the user;

and the sample image screening unit is used for calling a preset model to process the initial fundus image so as to screen out the sample fundus image which accords with the model training condition in the initial fundus image, wherein the sample fundus image is a real fundus image type and has an image quality larger than a quality threshold value.

Optionally, the sample image screening unit includes:

a preprocessing image generation subunit, configured to perform preprocessing on the initial fundus image, and generate a preprocessed fundus image;

the recognition result acquisition subunit is used for calling a fundus image recognition model to process the preprocessed fundus image so as to obtain a fundus image recognition result;

a standard image determining subunit configured to determine a standard fundus image in the initial fundus image according to the fundus image identification result;

the quality probability obtaining subunit is used for calling an image quality grade classification model to process the preprocessed fundus image so as to obtain the probability that the preprocessed fundus image belongs to each preset quality grade;

And the sample image screening subunit is used for screening sample fundus images meeting model training conditions from the standard fundus images according to the probability.

In a third aspect, an embodiment of the present application provides an electronic device, including:

a memory, a processor, and a computer program stored on the memory and executable on the processor, which when executed by the processor, implements the model training method of any of the above.

In a fourth aspect, embodiments of the present application provide a readable storage medium, which when executed by a processor of an electronic device, enables the electronic device to perform the model training method of any one of the above.

In the embodiment of the application, the sample fundus image conforming to the model training conditions of the user is obtained, and the sample fundus image is marked with the actual disease category of the Alzheimer's disease. Inputting a sample fundus image into a disease prediction model to be trained, wherein the disease prediction model to be trained comprises: feature screening, a blood vessel segmentation layer and a feature fusion layer. And calling a feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling treatment on the sample fundus image, so as to obtain a blood vessel labeling feature map. And calling a feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map, so as to obtain the predicted disease probability corresponding to the sample fundus image. And calculating a loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability. And under the condition that the loss value is in a preset range, obtaining a disease prediction model for predicting the Alzheimer disease. According to the embodiment of the application, the blood vessels in the fundus image are segmented and marked, so that the model can learn key clinical information (namely blood vessel segmentation and marking characteristics) required by the Alzheimer's disease more effectively, the interference of error information can be effectively avoided by screening the characteristics in the fundus image, the limitation that the traditional method is difficult to discover disease characteristics directly from the fundus image is overcome, and the prediction accuracy rate of the Alzheimer's disease can be improved greatly.

The foregoing description is only an overview of the technical solutions of the present application, and may be implemented according to the content of the specification in order to make the technical means of the present application more clearly understood, and in order to make the above-mentioned and other objects, features and advantages of the present application more clearly understood, the following detailed description of the present application will be given.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person of ordinary skill in the art.

FIG. 1 is a flowchart illustrating steps of a model training method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a disease prediction process according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a ResNet-50 network architecture according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a network incorporating a vessel segmentation and attention module according to an embodiment of the present application;

FIG. 5 is a schematic diagram of an attention module according to an embodiment of the present application;

Fig. 6 is a schematic structural diagram of a model training device according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

Referring to fig. 1, a flowchart illustrating steps of a model training method provided in an embodiment of the present application is shown, and as shown in fig. 1, the model training method may include: step 101, step 102, step 103, step 104, step 105 and step 106.

Step 101: and acquiring a sample fundus image of the user, which accords with the model training condition, wherein the sample fundus image is marked with the real disease category of the Alzheimer disease.

The embodiment of the application can be applied to the scene of training a model for predicting Alzheimer's disease.

The model training conditions refer to conditions in which fundus images of fundus image categories and image quality greater than a quality threshold are screened as model training samples.

The real disease category refers to a category manually marked on the sample fundus image, and in practical application, a designated number can be added to the image by a business person to indicate the real disease category of the sample fundus image, and the like.

When training a disease prediction model for alzheimer's disease, a sample fundus image of a user conforming to a model training condition can be acquired. Specifically, a plurality of fundus images of the user may be acquired, and then the plurality of fundus images may be first identified with a fundus image identification model to identify a standard fundus image of the plurality of fundus images. And finally, invoking an image quality grade classification model to classify the image quality of the identified standard fundus image, and screening out the standard fundus image with the highest quality to be used as a sample fundus image. The implementation may be described in detail in connection with the following specific implementations.

In a specific implementation of the present application, the step 101 may include:

substep A1: a plurality of initial fundus images of the user are acquired.

In the present embodiment, a plurality of initial fundus images of the user can be acquired at the time of training for the disease prediction model. In this example, the initial fundus image may be a fundus image of the user photographed by the medical institution, or may be a fundus image or the like photographed by itself by a specific fundus image photographing apparatus, and specifically, the manner of obtaining the initial fundus image may be determined according to the service requirement, which is not limited in this embodiment.

After a plurality of initial fundus images of the user are acquired, sub-step A2 is performed.

Substep A2: and calling a preset model to process the initial fundus image so as to screen out the sample fundus image which accords with the model training condition from the initial fundus image, wherein the sample fundus image is a real fundus image type and has an image quality greater than a quality threshold.

The sample fundus image may be a fundus image of a true fundus image category and an image quality greater than a quality threshold.

After a plurality of initial fundus images of the user are acquired, a preset model can be called to process the initial fundus images so as to screen out sample fundus images which accord with model training conditions in the initial fundus images. Specifically, the preset model may be: and the fundus image recognition model and the image quality grade classification model are used for cooperatively screening out a sample fundus image which accords with the model training condition in the initial fundus image. The implementation may be described in detail in connection with the following specific implementations.

In another specific implementation of the present application, the foregoing substep A2 may include:

Substep B1: and preprocessing the initial fundus image to generate a preprocessed fundus image.

In the present embodiment, after the initial fundus image is acquired, the initial fundus image may be preprocessed to generate a preprocessed fundus image. Specifically, the initial fundus image may be subjected to operations such as denoising, deduplication, and the like to obtain a preprocessed fundus image.

After the initial fundus image is preprocessed to generate a preprocessed image, sub-step B2 and sub-step B4 are performed.

Substep B2: and calling a fundus image recognition model to process the preprocessed fundus image, so as to obtain a fundus image recognition result.

After the initial fundus image is preprocessed to generate a preprocessed image, a fundus image recognition model can be called to process the preprocessed fundus image, and a fundus image recognition result is obtained. Specifically, the preprocessed image may be input to the fundus image recognition model to recognize the preprocessed fundus image by the fundus image recognition model, resulting in a fundus image recognition result.

And after invoking the fundus image recognition model to process the preprocessed fundus image to obtain a fundus image recognition result, executing a sub-step B3.

Substep B3: and determining a standard fundus image in the initial fundus image according to the fundus image identification result.

After the fundus image recognition model is called to process the preprocessed fundus image to obtain a fundus image recognition result, a standard fundus image in the initial fundus image can be determined according to the fundus image recognition result. Specifically, the non-fundus image in the initial fundus image, such as a face image or an image without fundus characteristics, can be removed through the fundus image recognition result, and the remaining initial fundus image is the standard fundus image after the non-fundus image in the initial fundus image is removed.

After the standard fundus image in the initial fundus image is determined from the fundus image recognition result, a sub-step B5 is performed.

Substep B4: and calling an image quality grade classification model to process the preprocessed fundus image to obtain the probability that the preprocessed fundus image belongs to each preset quality grade.

The preset quality level may be divided into three levels, respectively: a non-fundus image grade, a blurred fundus image grade, a high quality fundus image grade.

After the initial fundus image is preprocessed to generate a preprocessed image, an image quality grade classification model can be called to process the preprocessed fundus image, so that the probability that the preprocessed fundus image belongs to each preset quality grade is obtained. Specifically, the image quality class classification model may extract image features of the preprocessed fundus image, and classify the image features to obtain a probability that the preprocessed fundus image belongs to a preset quality class. The probability that the preprocessed fundus image belongs to the non-fundus image grade, the fuzzy fundus image grade and the high-quality fundus image grade can be output through the image quality grade classification model.

And B5, after invoking the image quality grade classification model to process the preprocessed fundus image to obtain the probability that the preprocessed fundus image belongs to each preset quality grade, executing the substep.

Substep B5: and screening sample fundus images meeting model training conditions from the standard fundus images according to the probability.

After the probability that the preprocessed fundus image belongs to each preset quality level is obtained, a sample fundus image conforming to the model training condition can be screened from the standard fundus image according to the probability that the obtained preprocessed fundus image belongs to each preset quality level. Specifically, the quality level to which the preprocessed fundus image belongs may be determined according to the probability of each preset quality level to which the preprocessed fundus image belongs, and then, a standard fundus image belonging to a high quality level is screened as a sample fundus image to perform a subsequent training process of the disease prediction model.

According to the method and the device for classifying the fundus image, the fundus image recognition model and the image quality class classification model are combined, so that more accurate fundus image quality classification can be obtained, interference of non-fundus images is filtered, accuracy of fundus image quality control is improved, and detection accuracy of a trained disease prediction model is improved.

In the present embodiment, the sample fundus image of a single user may be one image or may be a plurality of images. Specifically, the number of fundus images for a single user may be determined according to actual circumstances, which is not limited by the present embodiment.

After the sample fundus image of the user conforming to the model training conditions is acquired, step 102 is performed.

Step 102: inputting the sample fundus image to a disease prediction model to be trained, the disease prediction model to be trained comprising: feature screening, a blood vessel segmentation layer and a feature fusion layer.

After the sample fundus image of the user conforming to the model training conditions is acquired, the sample fundus image may be input to the disease prediction model to be trained. In this example, the disease prediction model to be trained may be ResNet-50. As shown in FIG. 3, the parameter quantity of the ResNet-50 is relatively low, and the ResNet structure can accelerate the training of the neural network very fast, so that the accuracy of the model is also greatly improved.

In this example, the disease prediction model to be trained may include: feature screening, a blood vessel segmentation layer and a feature fusion layer. The feature screening and vessel segmentation layer may include: an attention layer and a blood vessel segmentation layer.

Wherein the attention layer can be used to suppress information in the fundus image that is not related to Alzheimer's disease. The blood vessel segmentation layer can be used for enabling the network to make full use of blood vessel characteristics in fundus images to effectively classify the Alzheimer's disease. The feature fusion layer can fuse the feature graphs output by the two branches of the attention layer and the blood vessel segmentation layer.

After the sample fundus image is input to the disease prediction model to be trained, steps 103 and 104 are performed.

Step 103: and calling the feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling processing on the sample fundus image to obtain an attention feature map and a blood vessel labeling feature map.

After the sample fundus image is input into the disease prediction model to be trained, the feature screening and blood vessel segmentation layer can be called to carry out feature screening and blood vessel segmentation labeling treatment on the sample fundus image, and a blood vessel labeling feature map is obtained. Specifically, feature screening can be performed on the sample fundus image through the attention layer so as to inhibit irrelevant information in the sample fundus image, and further, effective information is fully expressed. And the blood vessels in the sample fundus image are segmented through the blood vessel segmentation layer, so that the blood vessel characteristics associated with the Alzheimer disease in the fundus image are fully utilized. The implementation may be described in detail in connection with the following specific implementations.

In a specific implementation of the present application, the step 103 may include:

substep S1: and calling the attention layer to perform feature screening processing on the sample fundus image, and outputting an attention feature map.

After the sample fundus image is input into the disease prediction model to be trained, the attention layer can be called to perform feature screening processing on the sample fundus image so as to inhibit information irrelevant to Alzheimer's disease in the sample fundus image and output an attention feature map. As shown in fig. 4, after the fundus image is input to the model, the fundus image may be processed by the encoder and then processed by the attention module, and the feature map output by the attention module is the attention feature map. The processing of the attention layer may be described in detail in connection with the following specific implementations.

In a specific implementation of the present application, the attention layer includes: the N convolutional layers, the feature map processing layer and the output layer, the parameters of the N convolutional layers are different, and the substep S1 may include:

substep C1: and calling the N convolution layers to respectively carry out channel compression processing on the sample fundus image, so as to generate N branch feature images.

In the embodiment of the application, after the sample fundus image is input to the disease prediction model to be trained, the sample fundus image can be subjected to coding processing through an encoder, namely, feature vectors with dimensions are set for feature coding bits in the sample fundus image through the encoder, and a coding feature map is output. As shown in fig. 4.

In fig. 4, the attention module is the attention layer in this example. The structure of the attention module may be as shown in fig. 5.

After the coding feature images output by the encoder are obtained, channel compression processing can be respectively carried out on the sample fundus images through N convolution layers, and N branch feature images are generated. As shown in fig. 5, n=3, three 1*1 convolutions with the same number are used for channel compression in this example, and the channel dimension is reserved to flatten the width and height into one dimension, which is mainly used herein to reduce the information redundancy of the input feature map, and reduce the complexity of the subsequent similarity calculation.

After the N branch feature maps are generated by performing channel compression processing on the sample fundus image by the N convolution layers, respectively, sub-step C2 is performed.

Substep C2: and calling the feature map processing layer to process the N branch feature maps to generate a fusion feature map.

After the N convolution layers respectively compress channels of the sample fundus image to generate N branch feature images, the feature image processing layer can be called to process the N branch feature images to generate a fusion feature image. The process may be described in detail in connection with the following specific implementations.

In another specific implementation of the present application, the foregoing substep C2 may include:

substep D1: and calling the feature map processing layer to carry out transposition operation on the first branch feature map to obtain a first transposition feature map.

In this embodiment of the present application, after obtaining three branch feature graphs, the feature graph processing layer may be called to transpose the first branch feature graph to obtain a first transpose feature graph.

After the feature map processing layer is called to transpose the first branch feature map to obtain a first transposed feature map, a sub-step D2 is performed.

Substep D2: and carrying out matrix multiplication on the first transfer characteristic diagram and the second branch characteristic diagram to obtain an intermediate characteristic diagram.

After the feature map processing layer is called to transpose the first branch feature map to obtain a first transposed feature map, the first transposed feature map and the second branch feature map can be subjected to matrix multiplication to obtain an intermediate feature map.

After matrix-multiplying the first transformed feature map with the second branched feature map to obtain an intermediate feature map, a sub-step D3 is performed.

Substep D3: and carrying out normalization processing on the intermediate feature map to obtain a normalized feature map.

After the first transfer feature map and the second branch feature map are subjected to matrix multiplication to obtain an intermediate feature map, normalization processing can be performed on the intermediate feature map to obtain a normalized feature map.

After the normalization processing is performed on the intermediate feature map to obtain a normalized feature map, a sub-step D4 is performed.

Substep D4: and carrying out matrix multiplication on the normalized feature map and the third branch feature map to obtain the fusion feature map.

After the normalized feature map is obtained by normalizing the intermediate feature map, the normalized feature map and the third branch feature map may be multiplied by a matrix to obtain a fused feature map.

For the implementation process, as shown in fig. 5, the feature diagrams of the output of the three convolution layers are respectively: f (x), g (x) and h (x). When the feature map processing is performed, the transpose operation can be performed on the feature map of the branch f (x) and then the matrix multiplication can be performed on the feature map of the branch g (x), and then the result is normalized through softmax. Finally, multiplying the normalized attention matrix by the feature map obtained by the branch h (x), namely, carrying out weight redistribution on different channels according to the similarity, obtaining the channel number by expanding the channel to the input feature map through softmax and 1*1 convolution, wherein key detail features in the output feature map are expressed more fully relative to the original feature map, so that attention weight distribution is realized.

After matrix multiplication of the normalized feature map and the third branch feature map to obtain a fused feature map, a sub-step C3 is performed.

Substep C3: and calling the output layer to process the fusion feature map and outputting the attention feature map.

After the normalized feature map and the third branch feature map are subjected to matrix multiplication to obtain a fused feature map, an output layer can be called to process the fused feature map, so that the attention feature map can be output.

Substep S2: and calling the blood vessel segmentation layer to perform blood vessel segmentation labeling processing on the sample fundus image, and outputting a blood vessel labeling feature map.

After the sample fundus image is input into the disease prediction model to be trained, a blood vessel segmentation layer can be called to carry out blood vessel segmentation labeling processing on the sample fundus image, and a blood vessel labeling feature map is output. In the specific implementation, the blood vessel in the sample fundus image can be segmented, and then the blood vessel labeling feature map can be obtained after pooling treatment. For this implementation, the detailed description may be made in connection with the following specific implementations.

In a specific implementation of the present application, the above sub-step S2 may include:

Substep E1: and calling the blood vessel segmentation layer to perform blood vessel segmentation labeling treatment on the blood vessel in the sample fundus image, so as to obtain a blood vessel segmentation feature map.

In the embodiment of the application, after the sample image is input into the disease prediction model to be trained, the blood vessel segmentation layer can be called to carry out blood vessel segmentation labeling processing on the blood vessels in the sample fundus image, so as to obtain a blood vessel segmentation feature map.

In specific implementation, previous researches show that arteriosclerosis is probably an independent risk factor of Alzheimer's disease of the elderly, and the eyeground examination shows that the arterial wall is thickened and hardened on an eyeground image and has uneven thickness, bent blood vessels and enhanced reflection of the arterial wall, and the serious case is in a copper wire appearance. Therefore, the blood vessel segmentation labels can be introduced and fused into the image features, so that the network can effectively classify the Alzheimer's disease by fully utilizing the blood vessel features in the fundus image.

And E2, after invoking the blood vessel segmentation layer to perform blood vessel segmentation labeling processing on the blood vessels in the sample fundus image to obtain a blood vessel segmentation feature map, executing the substep.

Substep E2: and carrying out pooling treatment on the blood vessel segmentation feature map, and outputting the blood vessel labeling feature map with the same dimension as the attention feature map.

After the blood vessel segmentation layer is called to carry out blood vessel segmentation labeling processing on blood vessels in the sample fundus image to obtain a blood vessel segmentation feature map, the blood vessel segmentation feature map can be subjected to pooling processing to output the blood vessel labeling feature map with the same dimension as the attention feature map. As shown in fig. 4.

After invoking the feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling on the sample fundus image to obtain a blood vessel labeling feature map, step 104 is executed.

Step 104: and calling the feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map to obtain the predicted disease probability corresponding to the sample fundus image.

After the blood vessel labeling feature map and the attention feature map are obtained, a feature fusion layer can be called to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map, so that the predicted disease probability corresponding to the sample fundus image is obtained. The process may be described in detail in connection with the following specific implementations.

In a specific implementation of the present application, the step 104 may include:

substep F1: and calling the feature fusion layer to perform matrix point multiplication processing on the attention feature map and the blood vessel labeling feature map to obtain a point multiplication feature map.

In the embodiment of the application, after the attention feature map and the blood vessel labeling feature map are obtained, the feature fusion layer can be called to perform matrix dot-multiplication processing on the attention feature map and the blood vessel labeling feature map, so that the dot-multiplication feature map can be obtained.

And after the feature fusion layer is called to perform matrix point multiplication processing on the attention feature map and the blood vessel labeling feature map to obtain a point multiplication feature map, executing a substep F2.

Substep F2: and performing cross multiplication processing on the point multiplication feature map and the attention feature map to obtain a target feature map.

After the feature fusion layer is called to perform matrix point multiplication processing on the attention feature map and the blood vessel labeling feature map to obtain a point multiplication feature map, cross multiplication processing can be performed on the point multiplication feature map and the attention feature map to obtain a target feature map

After performing the cross-multiplication process on the point-multiplication feature map and the attention feature map to obtain the target feature map, a sub-step F3 is performed.

Substep F3: and determining the predicted disease probability corresponding to the sample fundus image based on the target feature map.

After the cross multiplication processing is carried out on the point multiplication feature map and the attention feature map to obtain a target feature map, the predicted disease probability corresponding to the sample fundus image can be determined through the target feature map.

And after calling the feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map to obtain the predicted disease probability corresponding to the sample fundus image, executing step 105.

Step 105: and calculating a loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability.

And after invoking a feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map to obtain the predicted disease probability corresponding to the sample fundus image, calculating to obtain the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability labeled for the sample fundus image in advance. The specific computing process may be described in detail in connection with the following specific implementations.

In a specific implementation of the present application, the step 105 may include:

substep G1: and calculating a cross entropy loss value based on the real disease category and the predicted disease probability.

In the embodiment of the application, after the predicted disease probability is obtained, the cross entropy loss value can be calculated based on the real disease category and the predicted disease probability. Specifically, the calculation method can be shown in the following formula (1):

（1）

In the above-mentioned formula (1),for cross entropy loss value, < >>To predict disease probability>And labeling the true category.

After the cross entropy loss value is calculated based on the true disease category and the predicted disease probability, sub-step G2 is performed.

Substep G2: and calculating the loss value of the disease prediction model to be trained based on the number of samples in the round of training and the cross entropy loss value.

After the cross entropy loss value is calculated based on the real disease category and the predicted disease probability, the loss value of the disease prediction model to be trained can be calculated based on the number of samples of the round of training and the cross entropy loss value. Specifically, the calculation method can be shown in the following formula (2):

（2）

in the above-mentioned formula (2),for loss value, +_>Is the number of samples.

After calculating the loss value of the disease prediction model to be trained based on the true disease category and the predicted disease probability, step 106 is performed.

Step 106: and under the condition that the loss value is in a preset range, obtaining a disease prediction model for predicting Alzheimer's disease.

After calculating the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability, it may be determined whether the loss value is within a preset range.

If the loss value is not in the preset range, the disease prediction model to be trained is not converged, and at the moment, training can be continued on the model by combining more sample fundus images until the model converges.

If the loss value is within the preset range, the disease prediction model to be trained is converged, and the trained disease prediction model to be trained can be used as a final disease prediction model for predicting Alzheimer's disease.

According to the embodiment of the application, the blood vessels in the fundus image are segmented and marked, so that the model can learn key clinical information (namely blood vessel segmentation and marking characteristics) required by the Alzheimer's disease more effectively, the interference of error information can be effectively avoided by screening the characteristics in the fundus image, the limitation that the traditional method is difficult to discover disease characteristics directly from the fundus image is overcome, and the prediction accuracy rate of the Alzheimer's disease can be improved greatly.

After training to obtain a disease prediction model for predicting Alzheimer's disease, the model may be used to perform a predictive reasoning process for Alzheimer's disease. As shown in fig. 2, first, a plurality of sample fundus images of a person to be detected may be acquired, and the plurality of sample fundus images may be input to a quality control model to screen out fundus images with higher quality for prediction. After screening out the fundus image with higher quality, the fundus image obtained by screening may be input into the alzheimer's disease model (i.e., the disease prediction model in this example), so that a prediction result of whether the person to be detected has alzheimer's disease or not may be predicted.

According to the model training method provided by the embodiment of the application, the sample fundus image which accords with the model training conditions of the user is obtained, and the sample fundus image is marked with the real disease category of the Alzheimer's disease. Inputting a sample fundus image into a disease prediction model to be trained, wherein the disease prediction model to be trained comprises: feature screening, a blood vessel segmentation layer and a feature fusion layer. And calling a feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling treatment on the sample fundus image, so as to obtain a blood vessel labeling feature map. And calling a feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map, so as to obtain the predicted disease probability corresponding to the sample fundus image. And calculating a loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability. And under the condition that the loss value is in a preset range, obtaining a disease prediction model for predicting the Alzheimer disease. According to the embodiment of the application, the blood vessels in the fundus image are segmented and marked, so that the model can learn key clinical information (namely blood vessel segmentation and marking characteristics) required by the Alzheimer's disease more effectively, the interference of error information can be effectively avoided by screening the characteristics in the fundus image, the limitation that the traditional method is difficult to discover disease characteristics directly from the fundus image is overcome, and the prediction accuracy rate of the Alzheimer's disease can be improved greatly.

Referring to fig. 6, a schematic structural diagram of a model training apparatus provided in an embodiment of the present application is shown, and as shown in fig. 6, the model training apparatus 600 may include the following modules:

the sample image obtaining module 610 is configured to obtain a sample fundus image of a user according with a model training condition, where the sample fundus image is labeled with a real disease category of alzheimer's disease;

a sample image input module 620, configured to input the sample fundus image into a disease prediction model to be trained, where the disease prediction model to be trained includes: feature screening, a blood vessel segmentation layer and a feature fusion layer;

the labeling feature map obtaining module 630 is configured to invoke the feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling processing on the sample fundus image, so as to obtain an attention feature map and a blood vessel labeling feature map;

the prediction type obtaining module 640 is configured to invoke the feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map, so as to obtain a predicted disease probability corresponding to the sample fundus image;

a loss value calculation module 650, configured to calculate a loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability;

And the prediction model obtaining module 660 is configured to obtain a disease prediction model for predicting alzheimer's disease when the loss value is within a preset range.

the labeling feature diagram acquisition module comprises:

the attention attempt output unit includes:

the fusion feature map generation unit includes:

Optionally, the labeling feature map output unit includes:

Optionally, the prediction category acquisition module includes:

Optionally, the loss value calculation module includes:

Optionally, the sample image acquisition module includes:

Optionally, the sample image screening unit includes:

According to the model training device provided by the embodiment of the application, the sample fundus image which accords with the model training conditions of the user is obtained, and the sample fundus image is marked with the real disease category of the Alzheimer's disease. Inputting a sample fundus image into a disease prediction model to be trained, wherein the disease prediction model to be trained comprises: feature screening, a blood vessel segmentation layer and a feature fusion layer. And calling a feature screening and blood vessel segmentation layer to perform feature screening and blood vessel segmentation labeling treatment on the sample fundus image, so as to obtain a blood vessel labeling feature map. And calling a feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map, so as to obtain the predicted disease probability corresponding to the sample fundus image. And calculating a loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability. And under the condition that the loss value is in a preset range, obtaining a disease prediction model for predicting the Alzheimer disease. According to the embodiment of the application, the blood vessels in the fundus image are segmented and marked, so that the model can learn key clinical information (namely blood vessel segmentation and marking characteristics) required by the Alzheimer's disease more effectively, the interference of error information can be effectively avoided by screening the characteristics in the fundus image, the limitation that the traditional method is difficult to discover disease characteristics directly from the fundus image is overcome, and the prediction accuracy rate of the Alzheimer's disease can be improved greatly.

Additionally, the embodiment of the application also provides electronic equipment, which comprises: the system comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the computer program is executed by the processor to realize the model training method.

Fig. 7 shows a schematic structural diagram of an electronic device 700 according to an embodiment of the present invention. As shown in fig. 7, the electronic device 700 includes a Central Processing Unit (CPU) 701 that can perform various suitable actions and processes according to computer program instructions stored in a Read Only Memory (ROM) 702 or computer program instructions loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the electronic device 700 may also be stored. The CPU701, ROM702, and RAM703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

Various components in the electronic device 700 are connected to the I/O interface 705, including: an input unit 706 such as a keyboard, mouse, microphone, etc.; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, an optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information/data with other devices through a computer network, such as the internet, and/or various telecommunication networks.

The various procedures and processes described above may be performed by the processing unit 701. For example, the methods of any of the embodiments described above may be implemented as a computer software program tangibly embodied on a computer-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 700 via the ROM702 and/or the communication unit 709. When the computer program is loaded into RAM703 and executed by CPU701, one or more actions of the methods described above may be performed.

The embodiment of the application also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the processes of the above model training method embodiment, and can achieve the same technical effects, so that repetition is avoided, and no further description is given here. Wherein the computer readable storage medium is selected from Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic disk or optical disk.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), including several instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method described in the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of model training, the method comprising:

Calculating a loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability; the loss value is calculated based on the number of samples in the training round and a cross entropy loss value, and the cross entropy loss value is calculated based on the real disease category and the predicted disease probability;

obtaining a disease prediction model for predicting Alzheimer's disease under the condition that the loss value is in a preset range;

the feature screening and blood vessel segmentation layer comprises: an attention layer and a blood vessel segmentation layer,

invoking the blood vessel segmentation layer to perform blood vessel segmentation labeling processing on the sample fundus image, and outputting a blood vessel labeling feature map;

the invoking the feature fusion layer to perform feature fusion processing on the attention feature map and the blood vessel labeling feature map to obtain a predicted disease probability corresponding to the sample fundus image, including:

2. The method of claim 1, wherein the attention layer comprises: the device comprises N convolution layers, a feature map processing layer and an output layer, wherein parameters of the N convolution layers are different;

3. The method of claim 2, wherein the N branch feature graphs include, when n=3: a first branch feature map, a second branch feature map, and a third branch feature map,

normalizing the intermediate feature map to obtain a normalized feature map;

4. The method according to claim 1, wherein the invoking the vessel segmentation layer to perform vessel segmentation labeling processing on the sample fundus image, and outputting a vessel labeling feature map, includes:

5. The method according to claim 1, wherein the calculating a loss value of the disease prediction model to be trained based on the true disease category and the predicted disease probability comprises:

6. The method of claim 1, wherein the acquiring a sample fundus image of a user that meets model training conditions comprises:

acquiring a plurality of initial fundus images of the user;

7. The method of claim 6, wherein the invoking the pre-set model to process the initial fundus image to screen the sample fundus image of the initial fundus image that meets a model training condition comprises:

preprocessing the initial fundus image to generate a preprocessed fundus image;

8. A model training apparatus, the apparatus comprising:

The loss value calculation module is used for calculating the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease probability; the loss value is calculated based on the number of samples in the training round and a cross entropy loss value, and the cross entropy loss value is calculated based on the real disease category and the predicted disease probability;

the prediction model acquisition module is used for obtaining a disease prediction model for predicting Alzheimer's disease under the condition that the loss value is in a preset range;

the labeling feature diagram acquisition module comprises:

the labeling feature map output unit is used for calling the blood vessel segmentation layer to perform blood vessel segmentation labeling processing on the sample fundus image and outputting a blood vessel labeling feature map;

the prediction category acquisition module includes:

9. An electronic device, comprising:

memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor implements the model training method of any of claims 1 to 7.

10. A readable storage medium, characterized in that instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the model training method of any one of claims 1 to 7.