CN108647732B

CN108647732B - Pathological image classification method and device based on deep neural network

Info

Publication number: CN108647732B
Application number: CN201810456914.7A
Authority: CN
Inventors: 祝闯; 郭垚; 刘军; 刘芳
Original assignee: Beijing University of Posts and Telecommunications
Current assignee: Beijing University of Posts and Telecommunications
Priority date: 2018-05-14
Filing date: 2018-05-14
Publication date: 2020-07-31
Anticipated expiration: 2038-05-14
Also published as: CN108647732A

Abstract

The embodiment of the invention provides a pathological image classification method based on a deep neural network, which comprises the following steps: acquiring an image to be detected; processing the image to be detected to obtain a global image to be detected and a local image to be detected; predicting the global image to be detected and the local image to be detected aiming at each mixed dual model to obtain a global prediction result probability vector and a local prediction result probability vector; presetting the probability vector obtained by prediction to obtain a final local prediction result probability vector and a final global prediction result probability vector; performing fusion calculation on the probability vectors obtained by the processing according to a preset weight; adding the fused vectors to obtain a prediction vector; and determining the image type of the image to be detected according to the prediction vector and a preset rule. Compared with the prior art, the method and the device can extract more effective image features from the sample image, so that the neural network can be more fully learned, and the accuracy of the trained network model in image classification is improved.

Description

Pathological image classification method and device based on deep neural network

Technical Field

The invention relates to the technical field of computers, in particular to a pathological image classification method and device based on a deep neural network and electronic equipment.

Background

In the medical field, a pathologist is required to visually observe a pathological specimen under a microscope to diagnose a tumor, for example, a disease such as breast cancer. With the improvement of the health consciousness of the whole people, pathological specimens for health examination are increased year by year, so that the analysis work of the pathological specimens is greatly increased, and the number of required analyzers is greatly increased. Therefore, it is necessary to study a pathological image computer-aided classification method based on a tumor pathological image.

In the prior art, computer-aided classification methods for pathological images can be generally divided into two categories: the first is based on traditional machine learning methods; the second category is deep learning based methods. In the second method, a large number of precisely labeled pathological image samples are collected first. And then training a pre-designed deep neural network model based on the pathological image samples, and finally classifying the pathological images to be classified by using the trained deep neural network model.

In general, a pathological image has a large size and a high resolution, in which abundant global and local features are embedded. In contrast, since the conventional deep neural network has a deep layer containing a fully connected layer with a fixed vector length, which defines the size of an input image, when a pathological image sample is obtained, the pathological image is usually required to be reduced to a proper size, which results in the loss of local features of the pathological image. Or the pathological image is cut to a proper size, which brings about the problem of how to select the pathological image area with rich local features for cutting. Further, even if a pathological image region with rich local features is selected, the global features of the pathological image are lost. Therefore, the traditional deep neural network structure cannot fully extract effective information of pathological images, so that the deep neural network is insufficiently learned, and the accuracy of the existing deep learning-based method for classifying the pathological images is low.

Disclosure of Invention

The embodiment of the invention aims to provide a pathological image classification method and device based on a deep neural network and electronic equipment, so as to improve the accuracy of classification of pathological images to be detected. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides a pathological image classification method based on a deep neural network, where the method includes:

acquiring an image to be detected;

locally cutting the image to be detected according to a first preset size to obtain a plurality of cut images to be detected, wherein the cut images to be detected comprise all areas in the image to be detected;

reducing the image to be detected according to a second preset size to obtain a global image to be detected, and reducing the plurality of cut images to be detected according to the second preset size to obtain a plurality of local images to be detected;

inputting the local images to be measured into a local network model for prediction aiming at each of a plurality of mixed double models to obtain a local prediction result probability vector corresponding to each local image to be measured; inputting the global image to be detected into a global network model for prediction to obtain a global prediction result probability vector; each mixed dual-model comprises a local network model and a global network model, wherein the local network model is as follows: the method comprises the following steps of training a preset initial local depth neural network based on a local sample image and a corresponding class mark vector thereof, wherein the local sample image is obtained by: after locally cutting a pre-acquired sample image according to the first preset size, reducing the pre-acquired sample image according to the second preset size to obtain an image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same, and the global network model is as follows: training a preset initial global deep neural network based on the global sample image and the class label vector corresponding to the global sample image, wherein the global sample image is obtained by: reducing the sample image according to the second preset size to obtain an image;

for each of the multiple mixed dual models, presetting each obtained local prediction result probability vector to obtain a final local prediction result probability vector;

for each of the multiple mixed dual models, normalizing the obtained global prediction result probability vector to obtain a final global prediction result probability vector;

for each of the multiple mixed dual models, performing fusion calculation on the obtained final local prediction result probability vector and the final global prediction result probability vector according to a preset weight to obtain a fused vector;

adding a plurality of fused vectors corresponding to the plurality of mixed dual models to obtain a prediction vector;

and determining the image category of the image to be detected according to the prediction vector and a preset rule.

In a second aspect, an embodiment of the present invention provides a pathological image classification device based on a deep neural network, where the device includes:

the to-be-detected image acquisition module is used for acquiring an image to be detected;

the image cutting module is used for carrying out local cutting on the image to be detected according to a first preset size to obtain a plurality of cut images to be detected, wherein the cut images to be detected comprise all areas in the image to be detected;

the image reducing module is used for reducing the image to be detected according to a second preset size to obtain a global image to be detected, and reducing the plurality of cut images to be detected according to the second preset size to obtain a plurality of local images to be detected;

the detection vector acquisition module is used for inputting the local images to be detected into a local network model for prediction aiming at each of a plurality of mixed double models to obtain a local prediction result probability vector corresponding to each local image to be detected; inputting the global image to be detected into a global network model for prediction to obtain a global prediction result probability vector; each mixed dual-model comprises a local network model and a global network model, wherein the local network model is as follows: the model training module is obtained by training a preset initial local depth neural network based on a local sample image and a corresponding class mark vector thereof, wherein the local sample image is as follows: after locally cutting a pre-acquired sample image according to the first preset size, reducing the pre-acquired sample image according to the second preset size to obtain an image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same, and the global network model is as follows: the model training module is obtained by training a preset initial global deep neural network based on the global sample image and the class label vector corresponding to the global sample image, wherein the global sample image is as follows: reducing the sample image according to the second preset size to obtain an image;

the first result processing module is used for presetting each obtained local prediction result probability vector aiming at each of the plurality of mixed double models to obtain a final local prediction result probability vector;

the second result processing module is used for carrying out normalization processing on the obtained global prediction result probability vector aiming at each of the plurality of mixed double models to obtain a final global prediction result probability vector;

the fusion calculation module is used for performing fusion calculation on the obtained final local prediction result probability vector and the final global prediction result probability vector according to a preset weight aiming at each of the plurality of mixed double models to obtain a fused vector;

a predictive vector obtaining module, configured to add the multiple fused vectors corresponding to the multiple mixed dual models to obtain a predictive vector;

and the category determining module is used for determining the image category of the image to be detected according to the prediction vector and a preset rule.

In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;

a memory for storing a computer program;

the processor is configured to implement the method steps of any one of the above-mentioned pathological image classification methods based on the deep neural network according to the first aspect when executing the program stored in the memory.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which is a storage medium in a server, and a computer program is stored in the computer-readable storage medium, and when being executed by a processor, the computer program implements any one of the method steps in the deep neural network-based pathological image classification method provided in the first aspect.

As can be seen from the above, in the scheme provided in the embodiment of the present invention, after the image to be detected is obtained, the image to be detected is first locally cut to obtain a plurality of cut images to be detected, where the cut images to be detected include all areas of the image to be detected, and then the image to be detected and the plurality of cut images to be detected are reduced according to the second preset size to obtain the global image to be detected and the plurality of local images to be detected. And then, predicting the local image to be predicted and the global image to be predicted by using the local network model and the global network model which are included in each mixed dual model respectively to obtain a local prediction result probability vector and a global prediction result probability vector. Then, respectively carrying out presetting processing and normalization processing on the local prediction result probability vector and the global prediction result probability vector corresponding to each mixed dual model to obtain a final local prediction result probability vector and a final global prediction result probability vector, and carrying out fusion calculation on the final local prediction result probability vector and the final global prediction result probability vector according to a preset threshold. And finally, adding the vectors which correspond to the double models and are subjected to fusion calculation to obtain a prediction vector, and determining the image category of the image to be detected according to the prediction vector and a preset rule.

As can be seen from the above, in the embodiment of the present invention, when each hybrid dual model is trained, the global network model is trained by using the global sample image, and the local network model is trained by using the local sample image, because the global sample image includes the global features of the sample image and the local sample image includes the local features of the sample image, the deep neural network can learn the global features and the local features of the image sample. Furthermore, when a plurality of mixed dual models are trained, the deep neural network can learn the global features and the local features of more image samples. Furthermore, the images to be detected and the local images to be detected corresponding to the images to be detected are predicted by utilizing the plurality of mixed models, so that the accuracy of the image category of the images to be detected determined according to the obtained plurality of detection results can be greatly improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flowchart of a pathological image classification method based on a deep neural network according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of a hybrid dual-model training method according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of a pathological image classification device based on a deep neural network according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the prior art, the traditional deep neural network cannot sufficiently extract effective information of pathological images, so that the deep neural network is insufficiently learned, and the accuracy of the existing deep learning-based method for classifying the pathological images is low.

In order to solve the problems in the prior art, the embodiment of the invention provides a pathological image classification method and device based on a deep neural network and electronic equipment.

First, a method for classifying a pathological image based on a deep neural network according to an embodiment of the present invention is described below.

The pathological image classification method based on the deep neural network provided by the embodiment of the invention can be applied to any electronic equipment needing pathological image classification, such as a tablet computer, a processor and the like, and is not particularly limited herein, and is hereinafter referred to as electronic equipment for short.

As shown in fig. 1, a schematic flowchart of a pathological image classification method based on a deep neural network according to an embodiment of the present invention is provided, where the method includes:

s101: and acquiring an image to be detected.

S102: and locally cutting the image to be detected according to the first preset size to obtain a plurality of cut images to be detected.

And the plurality of cut images to be detected comprise all areas in the images to be detected.

S103: and reducing the image to be detected according to a second preset size to obtain a global image to be detected, and reducing a plurality of cut images to be detected according to the second preset size to obtain a plurality of local images to be detected.

S104: inputting a plurality of local images to be detected into a local network model for prediction aiming at each of a plurality of mixed double models to obtain a local prediction result probability vector corresponding to each local image to be detected; and inputting the global image to be detected into a global network model for prediction to obtain a global prediction result probability vector.

Wherein, each mixed dual model comprises a local network model and a global network model, and the local network model is as follows: the method is obtained by training a preset initial local depth neural network based on a local sample image and a corresponding class mark vector thereof, wherein the local sample image is as follows: after locally cutting a pre-acquired sample image according to a first preset size, reducing the image according to a second preset size to obtain an image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same, and the global network model is as follows: the method comprises the following steps of training a preset initial global deep neural network based on a global sample image and a class mark vector corresponding to the global sample image, wherein the global sample image is obtained by: reducing the sample image according to a second preset size to obtain an image;

s105: and for each of the plurality of mixed dual models, presetting each obtained local prediction result probability vector to obtain a final local prediction result probability vector.

S106: and aiming at each of the plurality of mixed dual models, normalizing the obtained global prediction result probability vector to obtain a final global prediction result probability vector.

S107: and aiming at each of the plurality of mixed dual models, performing fusion calculation on the obtained final local prediction result probability vector and the final global prediction result probability vector according to a preset weight to obtain a fused vector.

S108: and adding a plurality of fused vectors corresponding to the plurality of mixed dual models to obtain a prediction vector.

S109: and determining the image category of the image to be detected according to the prediction vector and a preset rule.

As can be seen from the above, in the embodiment of the present invention, when each hybrid dual model is trained, the global network model is trained using the global sample image, and the local network model is trained using the local sample image. Furthermore, when a plurality of mixed dual models are trained, the deep neural network can learn the global features and the local features of more image samples. Furthermore, the images to be detected and the local images to be detected corresponding to the images to be detected are predicted by utilizing the plurality of mixed models, so that the accuracy of the image category of the images to be detected determined according to the obtained plurality of detection results can be greatly improved.

In the step S101, the image to be measured refers to an image that needs to be subjected to image classification, and does not have any other limiting meaning. The image to be detected can be an image received by the electronic equipment and sent by equipment in communication connection with the electronic equipment, can also be an image stored in the electronic equipment, can also be an image acquired by the electronic equipment through image acquisition equipment of the electronic equipment, and the like.

The image under test acquired by the electronic device may have different sizes, for example, 2048 × 1536 pixels when the image under test is generated by magnifying a breast cancer tissue slice by a digital microscope.

In the embodiment of the invention, the preset initial local deep neural network for training the local network model generally has certain requirements on the size of the input image, and further, the trained local network model has the same requirements on the size of the input image. In step S101, the size of the image to be measured acquired by the electronic device is usually much larger than the required size of the image input by the local network model, and therefore, the image to be measured needs to be further processed so that the acquired local image to be measured can meet the requirement of the local network model on the size of the input image, that is, after the electronic device acquires the image to be measured, the steps S102 to S103 need to be performed to acquire the local image to be measured.

Similarly, the preset initial global depth neural network for training the global network model generally has a certain requirement on the size of the input image, and further, the electronic device needs to further process the image to be detected, so that the obtained global image to be detected can meet the requirement of the global network model on the size of the input image, that is, after the electronic device obtains the image to be detected, the step S103 needs to be executed to obtain the global image to be detected.

The electronic device executes the step S102, and may perform local cropping on the to-be-detected image according to a first preset size, so as to obtain a plurality of cropped to-be-detected images.

It should be noted that, the above-mentioned local cropping of the image to be detected may be local cropping without overlapping the image to be detected, that is, the obtained multiple cropped images to be detected all include a part of the content of the image to be detected, and at the same time, the content of each cropped image to be detected is different, or local cropping with overlapping the image to be detected, that is, the obtained multiple cropped images to be detected all include a part of the content of the image to be detected, and at the same time, the content of the cropped images to be detected may have the same part.

The electronic device may perform local cropping on the image to be detected by using any image processing method capable of performing local cropping on the image to be detected, for example, a Photoshop, a JPEG decoder (JPG image cropping tool), an ArcGIS image re-cropping tool, or the like may be used to perform local cropping on the image to be detected. The embodiment of the invention does not specifically limit the way of the electronic equipment for locally cutting the image to be detected.

After obtaining the plurality of cut images to be detected, the electronic device may execute step S103, that is, the image to be detected may be reduced according to a second preset size to obtain a global image to be detected, and the plurality of cut images to be detected may be reduced according to the second preset size to obtain a plurality of local images to be detected.

The electronic device can reduce the Image to be measured and cut the Image to be measured by using any Image processing method capable of reducing the Image to be measured and cut the Image to be measured, for example, tools such as Photoshop, Game Image, L bright Image Resizer and the like can be used for reducing the Image to be measured and cutting the Image to be measured.

The first preset size can be determined according to the precision requirement of image classification in practical application, the second preset size can be determined according to the requirement of the local network model on the size of the input image, and the first preset size is larger than the second preset size. The second preset size meets the requirement of the local network model on the size of the input image, so that the size of the local image to be detected obtained by the electronic equipment can meet the requirement of the local network model on the size of the input image.

Similarly, the second preset size may be determined according to a requirement of the global network model for the size of the input image, and the second preset size satisfies a limitation of the global network model for the size of the input image, so that the size of the global image to be measured obtained by the electronic device can satisfy the limitation of the global network model for the size of the input image. The application does not limit the specific numerical values of the first preset size and the second preset size.

For example, assuming that the initial local deep neural network and the initial global deep neural network are convolutional neural networks Goog L eNet, the first predetermined size may be 512 × 512 pixels, and the second predetermined size may be 224 × 224 pixels.

After the electronic device executes steps S102 to S103, the obtained global image to be measured includes all contents of the image to be measured, and each local image to be measured includes local contents of the image to be measured, so that the global image to be measured obtained by the electronic device is used for reflecting global features of the image to be measured, and each local image to be measured is used for reflecting partial local features of the image to be measured.

In order to ensure that the local network model can detect all local features of the image to be detected when predicting a plurality of local images to be detected, the local images to be detected need to include all regions in the image to be detected. Then, when the image to be detected is locally cut according to the first preset size, the plurality of cut images to be detected may include all the areas in the image to be detected, and thus the local image to be detected obtained by cutting the image to be detected in a reduced manner according to the second preset size also includes all the areas in the image to be detected. The plurality of cut images to be detected may be images with completely different contents, or may be images with a part of the same content, as long as it is ensured that the obtained plurality of cut images to be detected can include all areas in the images to be detected.

It should be noted that the size of the image to be detected is usually much larger than the size of the input image defined by the global network model and the local network model, so that if the electronic device directly performs local cropping on the image to be detected according to the second preset size and uses the image obtained by local cropping as the local image to be detected, the local content of the image to be detected included in the local image to be detected may be too small, and further, the local image to be detected may not reflect the local features of the image to be detected, or may only reflect the less effective local features of the image to be detected. Therefore, in order to ensure that enough effective local features of the image to be detected can be reflected in the local image to be detected, the electronic equipment firstly cuts the local image to be detected according to the first preset size to obtain a cut image to be detected, so that the enough effective local features of the image to be detected can be reflected in the cut image to be detected, and further, when the electronic equipment reduces the cut image to be detected according to the second preset size to obtain the local image to be detected, the local features of the enough effective local features of the image to be detected can be reflected in the local image to be detected.

For example, if the image to be detected is a pathological image of breast cancer, most local features of the image to be detected need to be reflected by the cell nucleus or cytoplasm of the cancer cell included in the image to be detected, and therefore, the local image to be detected of the image to be detected needs to include the complete cell nucleus or cytoplasm. In general, compared with the size of the image to be detected, the specified size of the local network model for the input image is smaller, when the electronic device directly cuts the image to be detected according to the second preset size, and the image obtained by local cutting is used as the local image to be detected, the local image to be detected may not include complete cell nucleus or cytoplasm, and further, the local image to be detected may not reflect the local feature of the image to be detected.

After obtaining the multiple local images to be measured and the global image to be measured, the electronic device may perform the step S104 for each of the multiple hybrid dual models, that is, the multiple local images to be measured are input into the local network model for prediction, and the local network model further outputs a local prediction result probability vector corresponding to each local image to be measured; and inputting the global image to be detected into a global network model, and outputting a global prediction result probability vector by the global network model.

The number of the mixed double models can be determined according to the precision requirement of pathological image classification in practical application, the number of the mixed double models is not specifically limited, and each mixed double model comprises a local network model and a global network model.

For each mixed dual model, when a local network model in the mixed dual model is trained, a local sample image is obtained by locally cutting a pre-obtained sample image according to a first preset size and then reducing the pre-obtained sample image according to a second preset size, and a local image to be detected by the local network model is obtained by locally cutting the obtained image to be detected according to the first preset size and then reducing the obtained image to be detected according to the second preset size, namely, the local sample image and the local image to be detected have the same obtaining method and size.

Furthermore, because the local network model is obtained by training a preset initial local deep neural network based on the local sample image and the class label vector corresponding to the local sample image, in the training process, the initial local deep neural network can learn the image characteristics of the local sample image, gradually establish the corresponding relationship between the image characteristics of the local sample image and the class label vector, and further obtain the local network model. Therefore, the trained local network model can be used for predicting the local image to be detected and outputting a local prediction result probability vector.

Wherein, each dimension in the category label vector has a corresponding relation with a preset image category, and the numerical value of the element corresponding to each dimension refers to: the probability that the image class of the sample image is the preset image class. Therefore, the numerical values of the elements corresponding to the dimensions in the output local prediction result probability vector refer to: and predicting the local image to be detected by the local network model to obtain the probability that the image category of the local image to be detected is the preset image category. Furthermore, the local prediction result probability vector may be used as an image category vector of the corresponding local to-be-measured image.

That is, in step S104, when the local network model predicts each local image to be detected, according to the image feature of each local image to be detected and the established correspondence between the image feature and the category label vector, the local prediction result probability vector of each local image to be detected is determined and output, and the electronic device can obtain the local prediction result probability vector corresponding to each local image to be detected, that is, the image category vector of the local image to be detected. Moreover, since the class label vectors corresponding to the local sample image and the corresponding sample image are the same, the local prediction result probability vector output by the local network model is the image class vector of the image to be measured corresponding to the local image to be measured.

When a global network model belonging to the same hybrid dual-model as the local network model is trained, the adopted global sample image is an image obtained by reducing the sample image according to a second preset size, and the global image to be detected used by the global network model is an image obtained by reducing the image to be detected according to the second preset size, that is, the global sample image and the global image to be detected have the same obtaining method and size.

Furthermore, because the global network model is obtained by training a preset initial global deep neural network based on the global sample image and the class label vector corresponding to the global sample image, in the training process, the initial global deep neural network can learn the image characteristics of the global sample image, gradually establish the corresponding relationship between the image characteristics of the global sample image and the class label vector, and further obtain the global network model. Therefore, the trained global network model can be used for predicting the global image to be detected and outputting a global prediction result probability vector.

As with the local predictor probability vector, the numerical values of the elements corresponding to the dimensions in the output global predictor probability vector also refer to: and predicting the global image to be detected by the global network model to obtain the probability that the image category of the global image to be detected is the preset image category. Furthermore, the global prediction result probability vector may be used as an image category vector of the corresponding global image to be measured.

That is, in step S104, when the global network model predicts the global image to be detected, the global prediction result probability vector of the global vector to be detected is determined and output according to the image features of the global detection image and the established correspondence between the image features and the category label vectors, and the electronic device may obtain the global prediction result probability vector of the global image to be detected, that is, the image category vector of the global image to be detected. Moreover, since the global sample image is an image obtained by reducing the sample image according to the second preset size, the category label vectors corresponding to the global sample image and the sample image are the same, and further, since the global image to be measured is an image obtained by reducing the image to be measured according to the second preset size, the global prediction result probability vector output by the global network model is also the image category vector of the image to be measured.

For clarity and clarity of the description, the hybrid dual-model training method will be described in the following.

For each hybrid dual model, after obtaining the plurality of local prediction result probability vectors, the electronic device may execute the step S105 to perform preset processing on each obtained local prediction result probability vector to obtain a final local prediction result probability vector. For clarity, a manner of performing preset processing on each obtained local prediction result probability vector by the electronic device to obtain a final local prediction result probability vector will be described in the following.

For each hybrid dual model, after obtaining the global prediction result probability vector, the electronic device may execute step S106 to perform normalization processing on the obtained global prediction result probability vector to obtain a final global prediction result probability vector.

For each hybrid dual model, after obtaining the final local prediction result probability vector and the final global prediction result probability vector, the electronic device may execute step S107, and perform fusion calculation on the obtained final local prediction result probability vector and the final global prediction result probability vector according to a preset weight to obtain a fused vector. For the sake of clear text, the method of obtaining the fused vector by performing fusion calculation on the obtained final local prediction result probability vector and the final global prediction result probability vector by the electronic device will be described in the following.

For example, if the size of the sample image is large and the resolution is low, the effect of the local features of the sample image reflected by the obtained local sample image is worse than the effect of the global features reflected by the obtained global sample image, so that the weight of the final local prediction result probability vector can be set to a smaller value, and the weight of the final global prediction result probability vector can be set to a larger value, so as to improve the accuracy of the obtained fused vector.

The method may also be determined according to the requirements of the detection accuracy of the local network model and the global network model in the hybrid dual model, for example, if the detection accuracy of the local network model in the hybrid dual model is higher and the detection accuracy of the global network model is lower than that of the local network model, the weight of the final local prediction result probability vector may be set to a larger value and the weight of the final global prediction result probability vector may be set to a smaller value, so as to improve the accuracy of the obtained fused vector.

Of course, the determination may also be performed according to the comprehensive conditions of the related information of the sample image and the requirements of the detection accuracy of the local network model and the global network model in the hybrid dual model, and the embodiment of the present invention does not limit the specific numerical value of the preset weight.

After obtaining the fused vector corresponding to each hybrid dual model, the electronic device may execute step S108 to add the obtained multiple fused vectors to obtain a prediction vector. And continuing to execute the step S109, and determining the image type of the image to be detected according to the prediction vector and the preset rule. For clarity of the text and the scheme, a manner of determining the image type of the image to be detected by the electronic device according to the prediction vector and the preset rule will be described in the following.

It should be noted that, in the above steps S102 to S104, for each mixed dual model, a local image to be measured and a global image to be measured may be obtained first, and then the local image to be measured is input into the local network model for prediction to obtain a local prediction result probability vector, and at the same time, the global image to be measured is input into the global network model for prediction to obtain a global prediction result probability vector; or obtaining a local image to be measured, inputting the local image to be measured into a local network model for prediction to obtain a local prediction result probability vector, obtaining a global image to be measured, and inputting the global image to be measured into the global network model for prediction to obtain a global prediction result probability vector; the local prediction result probability vector is obtained by inputting the local image to be predicted into the local network model. This is all reasonable.

In the above steps S104 to S107, for a plurality of hybrid dual models, the obtained local image to be measured and the obtained global image to be measured may be simultaneously input into a plurality of local network models and a plurality of global network models included in the plurality of hybrid models for prediction, so as to obtain a local prediction result probability vector and a global prediction result probability vector corresponding to each hybrid dual model, and then, for the local prediction result probability vector and the global prediction result probability vector corresponding to each hybrid dual model, the steps S105 to S107 are performed, so as to obtain a fused vector for each hybrid dual model; or according to the preset arrangement sequence of a plurality of mixed dual models, firstly inputting the obtained local image to be measured and the global image to be measured into a local network model and a global network model which are included in one mixed model for prediction to obtain a local prediction result probability vector and a global prediction result probability vector which correspond to the mixed dual models, calculating a fused vector which corresponds to the mixed dual models, then inputting the obtained local image to be measured and global image to be measured into local network model and global network model included in next mixed model to make prediction to obtain local prediction result probability vector and global prediction result probability vector corresponding to said next mixed double model, and the fused vector corresponding to the next hybrid dual model is calculated, so that the fused vector corresponding to each hybrid dual model can be obtained in sequence. This is all reasonable.

The above-mentioned hybrid dual-mode training mode is specifically described below, and the mode may be applied to any electronic device, for example, a tablet computer, a notebook computer, a mobile phone, and the like, and is not limited herein.

As an implementation manner of the embodiment of the present invention, as shown in fig. 2, a schematic flow chart of a hybrid dual-model training manner provided in the embodiment of the present invention is shown, and the manner may include:

s201: and constructing an initial global deep neural network and an initial local deep neural network.

Because each hybrid dual model comprises a global network model and a local network model, an initial global deep neural network and an initial local deep neural network are required to be constructed when each hybrid dual model is trained, wherein the initial global deep neural network and the initial local deep neural network can be convolutional neural networks with completely identical internal structures, such as Goog L eNet.

S202: a first preset number of images are acquired from a pre-acquired sample image set to serve as sample images, and other images in the image sample set serve as verification images.

The images in the sample image set may be from other electronic devices, which are communicatively connected to the electronic device for training the hybrid dual model and send the images to the electronic device, or may be from local, and the electronic device for training the hybrid dual model is acquired through its own image acquisition device.

It should be noted that, in order to ensure the accuracy of image classification, the images in the sample image set and the image to be detected generally adopt images acquired under the same scene, for example, if the image to be detected is a pathological image of breast cancer, the images in the sample image set may be pathological images of breast cancer at different stages, and if the image to be detected is a pathological image of tumor, the images in the sample image set may be pathological images of tumor of different types.

In order to ensure that the global network model and the local network model obtained by training can have higher prediction accuracy, images in a sample image set can be divided into sample images and verification images, the initial global deep neural network and the initial local deep neural network are trained by using the sample images to obtain the initial global network model and the initial local network model, the initial global network model and the initial local network model can be tested by using the verification images, whether parameters of the initial global deep neural network and the initial local deep neural network are adjusted or not is determined according to a test result, if parameters need to be adjusted, the initial global deep neural network and the initial local deep neural network after the parameters are adjusted are trained by using the sample images again after the adjustment, so that the model training is carried out through multiple parameter adjustments, and the initial global network model and the initial local network model obtained by final training can meet the precision requirement and can be determined as the global network model and the local network model.

In general, a first preset number of images in the sample image set may be acquired as sample images, and other images in the sample image set may be taken as verification images. In general, the first preset number may be determined according to the number of images in the sample image set and the number of mixed dual models, and when the number of images in the sample image set is M and the number of mixed dual models is N, the first preset number may be M

Of course, the first preset number may also be set in other manners, and the embodiment of the present invention does not limit the value and the setting manner of the first preset number.

In an embodiment of the step S202, when training different hybrid dual models, sample images may be sequentially acquired from the sample image set. That is, when training different hybrid dual models, the sample image set used for acquiring the sample images includes the same image, and further, when training different hybrid dual models, the same image may exist in the acquired sample images, and the same image may also exist in the verification image.

For example, assuming that the number M of images in the image sample set is 1000 and the number N of hybrid dual models is 5, the first preset number may be set by the above-described setting method, that is,

then in training the 5 hybrid dual models described aboveFor each of the above, 800 images may be obtained from 1000 images in the image sample set, and used as sample images corresponding to the mixed dual model, and the other 200 images in the image sample set may be used as verification images corresponding to the mixed dual model, and furthermore, a part of 800 sample images corresponding to each mixed dual model may be repeated, and a part of 200 verification images may also be repeated.

In another embodiment of the foregoing step S202, the images in the sample image set may be averagely grouped in advance according to the number of the mixed dual models, and the sample images in each group are randomly allocated, so that, when training the mixed dual models, images in several groups may be selected from the grouped images as sample images corresponding to the mixed dual models, images in other groups may be used as verification images corresponding to the mixed dual models, and for different mixed dual models, images in different groups may be selected as verification images, and further, when training different mixed dual models, the same image may exist in the acquired sample images, and the same image may not exist in the verification images.

For example, assuming that the number M of the image sample set is 1000, the number N of the mixed dual models is 5, and the 5 mixed dual models are respectively numbered as the 1 st, the 2 nd, the 3 rd, the 4 th and the 5 th, the 1000 images in the image sample set may be averagely divided into 5 groups, each group including 200 images, wherein the images in each group are randomly allocated, and the 5 groups of images are respectively numbered as the 1 st group, the 2 nd group, the 3 rd group, the 4 th group and the 5 th group.

Also, the first preset number may be set using the above-described setting method, that is,

then 800 images in sets 1-4 may be taken as sample images and 200 images in set 5 as verification images when training the 1 st hybrid dual model; in training the 2 nd hybrid dual model, 800 images in the 1 st to 3 rd groups and the 5 th group can be used as sample images, and 200 images in the 4 th group can be used as verification images; in training the 3 rd hybrid dual model, 800 images in the 1 st to 2 nd groups and 4 th to 5 th groups can be used as sample images, and 300 images in the 3 rd group can be used as verification images; in training the 4 th hybrid dual model, 800 images in the 1 st and 3-5 th groups may be used as sample images, and 200 images in the 2 nd group may be used as verification images; in training the 5 th hybrid bi-model, 800 images in sets 2-5 may be used as sample images and 200 images in set 1 may be used as verification images.

It should be noted that, in training each hybrid dual model, the sample image and the verification image used may be selected in other ways, but it is necessary to ensure that the sample image and the verification image used in training each hybrid dual model include all the images in the image sample set, and the verification images used in training each hybrid dual model are not the same.

Of course, in the step S202, the first preset number of images may be obtained from the preset sample image set in other manners to serve as the sample images, and the other images in the sample image set are taken as the verification images, which is not specifically limited in this application.

S203: and determining a corresponding class mark vector of each sample image.

After the sample images are obtained, the category label vector corresponding to each sample image is determined, and for the sake of clear lines, specific ways of determining the category label vector corresponding to each sample image will be exemplified in the following.

S204: and reducing the sample image and the verification image according to a second preset size to obtain a global sample image and a global verification image.

The constructed initial global depth neural network has certain requirements on the size of the input image, and the size of the sample image is usually much larger than the specified size of the image input by the initial global depth neural network, so that the sample image needs to be further processed so that the obtained global sample image can meet the requirements of the initial global depth neural network on the size of the input image, and therefore, after the sample image is obtained, step S204 needs to be performed to reduce the sample image according to a second preset size to obtain a global sample image.

The initial global network model is obtained by training the initial global deep neural network, so that the initial global network model has the same requirement on the size of the input image, and further, when the initial global network model obtained by training is tested by using the global verification image, the predicted image needs to be reduced according to a second preset size, so that the obtained global verification image meets the requirement of the initial global network model on the size of the input image.

After the prediction accuracy of the initial global network model obtained by training meets the requirement, the initial global network model can be determined as the global network model, so that when the images to be detected are classified by using the global network model, the images to be detected also need to be reduced according to a second preset size, so that the obtained global images to be detected meet the requirement of the global network model on the size of the input images.

S205: and inputting the global sample image and the corresponding category label vector thereof into an initial global deep neural network for training to obtain an initial global network model.

In the training process, the initial global deep neural network can learn the image characteristics of each global sample image, output the category label vector corresponding to each global sample image, and establish the corresponding relation between the image characteristics and the category label vector through the learning of a large number of global sample images, so as to obtain an initial global network model.

It should be noted that the image features learned by the initial global deep neural network in the training process are global features of the sample images, that is, the global sample images corresponding to the initial global deep neural network include all contents of the sample images, and reflect the global features of the sample images.

S206: inputting the global verification image into an initial global network model for prediction to obtain initial global prediction accuracy;

after an initial global network model is obtained through training, in order to verify whether the prediction precision of the initial global network model meets requirements or not, the obtained global verification images can be input into the initial global network model for prediction, a global test result vector of each global verification image is obtained, the global test result vector is compared with a category label vector corresponding to each global verification image, whether the prediction result of the initial global network model on the global verification images is correct or not is judged, and then the ratio of the number of the global verification images with correct prediction results to the number of the global verification images can be calculated, and the initial global prediction accuracy is obtained.

For example, if the number of global verification images corresponding to a hybrid dual model is 200 and the number of global verification images with correct prediction results is 195, the initial global prediction accuracy is 195/200 × 100% to 97.5% can be calculated.

It should be noted that, in the embodiment of the present invention, in order to facilitate the calculation of the initial global prediction accuracy, the category label vector corresponding to each verification image may be determined at the same time when the step S203 is performed to determine the category label vector corresponding to each sample image. Each image in the sample image set needs to participate in the training process of the hybrid dual model, so that the category label vector corresponding to each image in the sample image set can be determined simultaneously when the sample image set is obtained in advance. The above-mentioned manner for determining the category label vector corresponding to the verification image and the manner for determining the category label vector corresponding to each image in the sample image set may be the same as the manner for determining the category label vector corresponding to each sample image in step S203.

S207: judging whether the absolute value of the first difference value is not larger than a preset global model threshold value,

wherein the first absolute difference value is: the absolute value of the difference between the output result accuracy of the initial global network model and the initial global prediction accuracy;

if the first difference absolute value is not greater than the global model threshold, step S208 is performed, and if the first difference absolute value is greater than the global network model threshold, step S209 is performed.

S208: stopping training, and determining the initial global network model as a global network model;

s209: adjusting parameters of the initial global deep neural network, and executing step S205;

after the global sample image and the corresponding class label vector thereof are input into the initial global deep neural network for training to obtain an initial global network model, the global sample image can be predicted by using the initial global network model obtained by training to obtain an output result of the initial global network model, that is, the output result of the initial global network model is a class test vector obtained by predicting the global sample image by using the initial global network model. Then, the category label vector and the category test vector corresponding to each global sample image can be compared, and the accuracy of the category test vector relative to the category label vector is calculated, so that the accuracy of the output result of the initial global network model is obtained.

For example, the number of the global sample images is 200, where the category label vectors corresponding to 194 global sample images are the same as the image categories of the global sample images determined by the category test vector, and then the accuracy of the category test vector with respect to the category label vectors can be calculated to be 97%, that is, the accuracy of the output result of the initial global network model is 97%.

Since the initial global deep neural network model may be overfitting to the model, too many learned features, and the like when the global features of the sample image are learned, which in turn causes a discrepancy between its output result accuracy and the initial global prediction accuracy, and, therefore, after the initial global network model is obtained by training, the initial global network model can be verified by utilizing the prediction image, the absolute value of the difference between the accuracy of the output result of the initial global network model obtained by training and the initial global prediction accuracy is calculated, hereinafter referred to as the first difference absolute value, and according to the magnitude relation between the first difference absolute value and the preset global model threshold, it is determined whether the parameter adjustment of the initial global deep neural network model is needed to be continued, and further, and training the initial global deep neural network model after parameter adjustment again to obtain a new initial global network model.

It should be noted that the preset global model threshold may be determined according to the accuracy requirement for image classification in practical application, and the higher the accuracy requirement for image classification is, the lower the preset global model threshold may be, and the specific value of the preset global model threshold is not limited in the present application.

If the first absolute value of the difference is not greater than the preset global model threshold, it is determined that the deviation between the accuracy of the output result of the initial global network model and the initial global prediction accuracy is small, and the initial global network model can obtain a high prediction accuracy when predicting the image to be detected, that is, after step S207 is executed, step S208 is continuously executed.

When the first absolute difference value is greater than the global model threshold, it indicates that the deviation between the output result accuracy of the initial global network model and the initial global prediction accuracy is large, and the prediction accuracy obtained when the initial global network model is used to predict the image to be measured is low, so that the initial global deep neural network needs to be subjected to parameter adjustment, and the initial global deep neural network model after parameter adjustment is retrained again to obtain a new initial global network model, that is, after step S207 is executed, step S209 is continuously executed.

For example, if the accuracy of the output result of the initial global network model is 98.2%, the accuracy of the initial global prediction is 97.4%, and the preset global model threshold is 0.5%, the first absolute difference value is 0.8%, and since 0.8% > 0.5%, the first absolute difference value is greater than the global model threshold, and then step S209 is continuously executed.

For another example, if the accuracy of the output result of the initial global network model is 98.2%, the accuracy of the initial global prediction is 98.4%, and the preset global model threshold is 0.5%, the first absolute difference value is 0.2%, and since 0.2% < 0.5%, the first absolute difference value is not greater than the global model threshold, and then step S208 is continuously executed. After the global network model is finally obtained, the global network model can be used for predicting the global image to be detected and outputting a global prediction result probability vector corresponding to the global image to be detected, wherein the global prediction result probability vector is the image category vector of the global image to be detected.

In one implementation, the initial global network model trained in step S205 may be a network model obtained when the iteration number of the initial global deep neural network reaches a preset number or when the output accuracy of the obtained initial global network model reaches a preset accuracy, and then step S206 and step S207 are executed;

in another implementation, the initial global network model trained in step S205 may be a network model obtained after each iteration of the initial global deep neural network model, that is, after each iteration of the initial global deep neural network model, step S206 and step S207 may be executed, and whether to stop the next iteration is determined according to the determination result of step S207, so as to obtain the global network model.

S210: performing local cropping according to a first preset size for each sample image and each verification image, respectively, and respectively obtaining a second preset number of local sample cropped images and local verification cropped images,

and the local sample trimming image and the corresponding sample image have the same category mark vector.

It should be noted that, the above-mentioned local cropping of the sample image and the verification image may be the local cropping without overlapping the sample image and the verification image, that is, the obtained multiple local sample cropped images all include a part of the content of the sample image, and at the same time, the content of each local sample cropped image is different, the obtained multiple local verification cropped images all include a part of the content of the verification image, and at the same time, the content of each local verification cropped image is different, or the sample image and the verification image are subjected to the local cropping with overlapping, that is, the obtained multiple local sample cropped images all include a part of the content of the sample image, and at the same time, the content of the local sample cropped images may have the same part, and the obtained multiple local verification cropped images all include a part of the content of the sample image, while the content of these test sample cropped images may exist in the same portion.

The sample image and the verification image may be locally cropped using any image processing method capable of locally cropping the sample image and the verification image, for example, a Photoshop, a JPEG repeat (JPG, image cropping tool), an ArcGIS image re-cropping tool, or the like may be used to locally crop the sample image and the verification image. The embodiment of the invention does not specifically limit the way of locally cutting the sample image and the verification image.

The second preset number can be determined according to the characteristics of the images in the sample image set and the accuracy requirement of image classification in practical application, the higher the accuracy requirement of image classification is, the larger the second preset number can be, and the specific numerical value of the second preset number is not limited in the application.

S211: reducing the obtained local sample trimming image and the local verification trimming image according to a second preset size to obtain a local sample image and a local verification image,

and the local sample image and the corresponding sample image have the same corresponding category label vector.

The sample Image, the partial sample cropped Image, and the partial verification cropped Image may be reduced using any Image processing method capable of reducing the sample Image, the partial sample cropped Image, and the partial verification cropped Image, for example, Photoshop, Game Image, L light Image Resizer, and the like may be used to reduce the sample Image, the partial sample cropped Image, and the partial verification cropped Image.

The first preset size can be determined according to the precision requirement of image classification in practical application, the second preset size can be determined according to the internal structure of the initial local depth neural network, the first preset size is larger than the second preset size, the second preset size meets the requirement of the initial local depth neural network on the size of the input image, and the size of the obtained local sample image can meet the requirement of the initial local depth neural network on the size of the input image.

It should be noted that, in practical applications, the size of the sample image is usually much larger than the required size of the input image defined by the initial local depth neural network, and therefore, if the sample image is locally cropped according to the second preset size directly and the image obtained by local cropping is used as the local sample image, the local content of the sample image included in the local sample image may be too small, and further, the local sample image may not reflect the local features of the sample image or may only reflect the less effective local features of the sample image. Therefore, in order to ensure that the local sample image can reflect the local features of enough effective sample images, after the sample image is obtained, local cropping can be performed according to a first preset size, and then the obtained local sample cropped image is reduced according to a second preset size to obtain the local sample image, so that the local sample image can reflect the local features of the enough effective sample images.

The local sample trimming image is an image obtained by locally trimming a sample image according to a first preset size, and the local sample image is an image obtained by reducing the local sample trimming image according to a second preset size, so that the class mark vectors corresponding to the local sample trimming image and the corresponding sample image are the same, and further, the class mark vectors corresponding to the local sample image and the corresponding sample image are also the same.

The initial local network model is obtained by training the initial local depth neural network, so that the initial global network model has the same requirements on the size of an input image and the included local features, further, when the initial local network model obtained by training is tested by using the local verification image, the verification image needs to be locally cut according to a first preset size, and then the obtained local verification cut image is reduced according to a second preset size to obtain the local verification image, so that the local verification image can meet the requirements of the initial local network model on the size of the input image and the reported local features.

After the prediction accuracy of the initial local network model obtained by training meets the requirement, the initial local network model can be determined as the local network model, so that when the local network model is used for classifying the image to be detected, the image to be detected needs to be locally cut according to a first preset size, and the obtained cut image to be detected is reduced according to a second preset size, so that the obtained local image to be detected meets the requirements of the local network model on the size of the input image and the local characteristics of the reporting examination.

S212: and inputting the local sample image and the corresponding class mark vector thereof into an initial local depth neural network for training to obtain an initial local network model.

In the training process, the initial local depth neural network can learn the image characteristics of each local sample image, output the category label vector corresponding to each local sample image, and establish the corresponding relation between the image characteristics and the category label vector through the learning of a large number of local sample images, so as to obtain an initial local network model.

It should be noted that the initial local deep neural network may learn local features of the image during the training process, that is, the local sample image corresponding to the initial local deep neural network includes local content of the sample image, which reflects local information features of the sample image. The local sample image corresponding to the same sample image in the plurality of local sample images may include all regions of the sample image, or may include a partial region of the sample image, where the included partial region is a region where the local features in the sample image are the most significant and most effective, and therefore, although the local sample image does not include all regions of the sample image, the local sample image may still reflect the local features of the sample image that are sufficiently significant.

S213: inputting the local verification image into an initial local network model for prediction to obtain an initial local prediction accuracy;

after the initial local network model is obtained through training, in order to verify whether the prediction precision of the initial local network model meets requirements or not, the obtained local verification images can be input into the initial local network model for prediction, a local test result vector of each local verification image is obtained, the local test result vector is compared with a category label vector corresponding to the local verification image, whether the prediction result of the initial local network model on the local verification images is correct or not is judged, and then the ratio of the number of the local verification images with correct prediction results to the number of the local verification images can be calculated, and the initial local prediction accuracy is obtained.

The local verification image is obtained by reducing the local verification trimming image according to the second preset size, and the local verification trimming image is obtained by locally trimming the verification image according to the first preset size, so that the category mark vector of the local verification image is the same as the category mark vector of the corresponding verification image.

For example, if the number of local verification images corresponding to a hybrid dual model is 200 and the number of local verification images with correct prediction results is 187, the initial local prediction accuracy is calculated to be 187/200 × 100% to 93.5%.

It should be noted that, corresponding to the local sample image, the local verification image includes the local content of the verification image, and reflects the local information feature of the verification image. The local verification image corresponding to the same verification image in the multiple local verification images may include all regions of the verification image, or may include partial regions of the verification image, where the partial regions are regions where the local features in the verification image are the most and most effective, so that although the local verification image does not include all regions of the verification image, the local test image may still reflect the local features of the verification image that are sufficiently effective.

In the embodiment of the present invention, in order to facilitate the calculation of the initial local prediction accuracy, the class mark vector corresponding to each verification image may be determined at the same time when the step S203 is performed to determine the class mark vector corresponding to each sample image. Each image in the sample image set needs to participate in the training process of the hybrid dual model, so that the category label vector corresponding to each image in the sample image set can be determined simultaneously when the sample image set is obtained in advance. The above-mentioned manner for determining the category label vector corresponding to the verification image and the manner for determining the category label vector corresponding to each image in the sample image set may be the same as the manner for determining the category label vector corresponding to each sample image in step S203.

S214: judging whether the absolute value of the second difference value is not larger than a preset local model threshold value,

wherein the second absolute difference value is: the absolute value of the difference between the output result accuracy of the initial local network model and the initial local prediction accuracy;

if the second difference absolute value is not greater than the local model threshold, step S215 is performed, and if the second difference absolute value is greater than the local network model threshold, step S216 is performed.

S215: stopping training, and determining the initial local network model as a local network model;

s216: and adjusting parameters of the initial local deep neural network, and executing the step S212.

After the local sample image and the corresponding class label vector thereof are input into the initial local deep neural network to be trained to obtain an initial local network model, the local sample image can be predicted by using the initial local network model obtained by training to obtain an output result of the initial local network model, that is, the output result of the initial local network model is a class test vector obtained by predicting the local sample image by using the initial local network model. Then, the category label vector and the category test vector corresponding to each local sample image can be compared, and the accuracy of the category test vector relative to the category label vector is calculated, so that the accuracy of the output result of the initial local network model is obtained.

For example, the number of the local sample images is 600, where the category label vectors corresponding to 594 local sample images are the same as the image categories of the local sample images determined by the category test vector, and then the accuracy of the category test vector with respect to the category label vector can be calculated to be 99%, that is, the accuracy of the output result of the initial local network model is 99%.

Since the initial local deep neural network model may be over-fitted when learning the local features of the sample image, too many learned features, and the like, thereby causing a deviation between the accuracy of the output result and the initial local prediction accuracy, and therefore, after the initial local network model is obtained by training, the initial local network model can be verified by utilizing the prediction image, the difference absolute value between the accuracy of the output result of the initial local network model obtained by training and the initial local prediction accuracy is calculated, the second difference absolute value is called as the second difference absolute value hereinafter, and whether the parameter adjustment of the initial local deep neural network model is needed to be carried out continuously is judged according to the magnitude relation between the second difference absolute value and the preset local model threshold value, and further, and training the initial local deep neural network model after parameter adjustment again to obtain a new initial local network model.

It should be noted that the preset local model threshold may be determined according to the accuracy requirement for image classification in practical application, and the higher the accuracy requirement for image classification is, the lower the preset local model threshold may be, and the specific numerical value of the preset local model threshold is not limited in the present application.

Whether the second difference absolute value is not greater than the preset local model threshold or not can be judged, when the second difference absolute value is not greater than the local model threshold, it is indicated that the deviation between the output result accuracy of the initial local network model and the initial local prediction accuracy is small, the initial local network model can obtain a high prediction accuracy when predicting the image to be detected, that is, after S214 is executed, S215 is continuously executed.

When the second difference absolute value is greater than the local model threshold, it indicates that the deviation between the output result accuracy of the initial local network model and the initial local prediction accuracy is large, and the prediction accuracy obtained when the initial local network model is used to predict the image to be measured is low, so that the initial local deep neural network needs to be subjected to parameter adjustment, and the initial local deep neural network model after parameter adjustment is retrained again to obtain a new initial local network model, that is, after S214 is executed, S216 is continuously executed.

For example, if the accuracy of the output result of the initial local network model is 96.2%, the accuracy of the initial local prediction is 95.4%, and the preset local model threshold is 0.5%, the second absolute difference value is 0.8%, and since 0.8% > 0.5%, the second absolute difference value is greater than the local model threshold, and then step S216 is continuously executed.

For another example, if the accuracy of the output result of the initial local network model is 99.2%, the accuracy of the initial local prediction is 99.4%, and the threshold of the preset local model is 0.5%, the absolute value of the second difference is 0.2%, and since 0.2% < 0.5%, the absolute value of the second difference is not greater than the threshold of the local model, the step S215 is further performed.

After the local network model is finally obtained, the local network model can be used for predicting the local image to be detected and outputting a local prediction result probability vector corresponding to the local image to be detected, wherein the local prediction result probability vector is the image category vector of the local image to be detected.

In one implementation, the initial local network model obtained by training in step S212 may be a network model obtained when the number of iterations of the initial local deep neural network reaches a preset number or when the output accuracy of the obtained initial local network model reaches a preset accuracy, and then S213 and S214 are executed;

in another implementation manner, the initial local network model trained in step S212 may be a network model obtained after each iteration of the initial local deep neural network model, that is, after each iteration of the initial local deep neural network model, S213 and S214 may be executed, and whether to stop the next iteration is determined according to the determination result of S214 to obtain the local network model.

It should be noted that, when training a plurality of hybrid dual models, after a hybrid dual model is obtained by training, another hybrid dual model is started to be trained until the number of the hybrid dual models obtained by training satisfies the predetermined number of the hybrid dual models, a local sample image and a global template image are obtained for each hybrid dual model, and a plurality of hybrid dual models are simultaneously trained, or a local sample image is obtained for each hybrid dual model, a local network model in each hybrid dual model is trained at the same time, a global sample image is obtained, and a global network model in each hybrid dual model is trained at the same time. The embodiment of the invention does not specifically limit the training sequence of a plurality of mixed dual models.

The following describes a specific way of determining the category label vector corresponding to each sample image in step S203. As an implementation manner of the embodiment of the present invention, the step S203 may include:

a1: and determining the probability that the image category of each sample image is a preset image category, wherein the preset image categories are multiple.

The image category can be determined according to the classification requirement of the image to be detected in practical application.

For example, when the image to be detected is a pathological breast cancer image, the classification requirement of the image to be detected is to determine which type of breast cancer the breast cancer corresponding to the image to be detected is, because the breast cancer can be generally classified into four categories, namely, a benign tumor, an in-situ tumor, an early tumor and an invasive tumor, when the embodiment of the present invention is applied to classification of pathological breast cancer images, the preset image categories may be the four categories of breast cancer.

The image category can also be determined according to the precision requirement of the classification of the image to be measured in practical application. For example, when the image to be detected is a tumor pathological image, if the requirement on the classification accuracy of the image to be detected is high, it is necessary to determine which tumor the tumor pathological image corresponds to, and also to determine which development stage the tumor pathological image corresponds to, then the image to be detected may be classified according to different development stages of different types of tumors, for example, in-situ tumor of breast cancer, infiltration tumor of breast cancer, squamous cell carcinoma tumor of lung cancer, adenocarcinoma tumor of lung cancer, and the like. If the classification accuracy requirement of the image to be detected is low, and only the tumor of which the tumor pathological image corresponds to is determined, the classification of the image to be detected can be performed according to different types of tumors, for example, breast cancer tumors, lung cancer tumors, liver cancer tumors, and the like.

It should be noted that the preset image categories are usually plural. In a very special case, the preset image category may be "target type" or "non-target type", that is, in this case, the essence of classifying the image to be tested is to determine whether the type to be tested belongs to the target type.

After the sample images are acquired, the probability that the image class of each sample image is a preset image class may be determined. For example, the probability that the image class of the sample image is the preset image class may be determined by analyzing the sample image. After the probabilities are determined, the correspondence between the acquired probabilities and the sample images may be recorded.

A2: and setting a category vector, wherein the dimension of the category vector is the same as the number of preset image categories.

It should be noted that the initial value of the element corresponding to each dimension in the set category vector may be any value, that is, the initial value of the set category vector may be any value. For example, assuming that the number of preset image categories is 4, the set category vector is a four-dimensional vector, and it is reasonable that the initial value of the set category vector may be [0,0,0,0], or [1,2,3,4], or [2,4,6,8 ].

A3: and determining the probability corresponding to the element corresponding to each dimension in the category vector according to the corresponding relation between the preset dimension and the preset image category and the determined probability.

After determining the probability that the image category of each sample image is the preset image category and setting the category vector, the probability corresponding to each dimension element in the set category vector can be determined according to the corresponding relationship between the dimension of the preset category vector and the preset image category and the determined probability.

For example, taking a sample image as an example, assuming that the preset image categories are A, B, C, D respectively, the dimension of the prediction vector is 4, and assuming that the corresponding relationship between the preset dimension and the preset image category is: the first dimension corresponds to a preset image category A, the second dimension corresponds to a preset image category B, the third dimension corresponds to a preset image category C, and the fourth dimension corresponds to a preset image category D. In step a1, the determined probabilities that the image category of the sample image is the preset image category are respectively: the probability of being image class a is 0, the probability of being image class B is 0.1, the probability of being image class C is 0.2, and the probability of being image class D is 0.7.

Then, it may be determined that the probability corresponding to the element corresponding to each dimension in the category vector corresponding to the sample image is: the probability corresponding to the element corresponding to the first dimension is 0, the probability corresponding to the element corresponding to the second dimension is 0.1, the probability corresponding to the element corresponding to the third dimension is 0.2, and the probability corresponding to the element corresponding to the fourth dimension is 0.7.

A4: and determining the numerical value of each element according to the determined probability corresponding to the element corresponding to each dimension to obtain the category label vector of each sample image.

After the probability corresponding to the element corresponding to each dimension in the category vector corresponding to each sample image is determined, the numerical value of each element in the category vector can be determined according to the determined probability corresponding to the element corresponding to each dimension, and then the category label vector of each sample image is obtained. For example, when the probability value corresponding to an element corresponding to a certain dimension in the category vector corresponding to the determined sample image is the maximum, the value of the element corresponding to the dimension is set to be a first preset value, and the values of the elements corresponding to other dimensions are set to be second preset values, where the first preset value may be 1, and the second preset value may be 0. This is all reasonable.

For example, taking a sample image as an example, assuming that the preset image categories are A, B, C, D respectively, the dimension of the prediction vector is 4, and the probability corresponding to the element corresponding to each dimension in the category vector corresponding to the sample image is determined as follows: the probability corresponding to the element corresponding to the first dimension is 0, the probability corresponding to the element corresponding to the second dimension is 0.1, the probability corresponding to the element corresponding to the third dimension is 0.2, and the probability corresponding to the element corresponding to the fourth dimension is 0.7.

Furthermore, it may be determined that the numerical value of the element corresponding to each dimension in the category vector corresponding to the sample image is: the numerical value of the element corresponding to the first dimension is 0, the numerical value of the element corresponding to the second dimension is 0.1, the numerical value of the element corresponding to the third dimension is 0.2, and the numerical value of the element corresponding to the fourth dimension is 0.7. I.e., the category label vector is [0,0.1,0.2,0.7 ].

Of course, the numerical values of the elements corresponding to the dimensions in the category vector corresponding to the sample image may also be determined as follows: the numerical value of the element corresponding to the first dimension is 0, the numerical value of the element corresponding to the second dimension is 0, the numerical value of the element corresponding to the third dimension is 0, and the numerical value of the element corresponding to the fourth dimension is 1. I.e., the category label vector is [0,0,0,1 ].

The following describes a specific manner of performing preset processing on each obtained local prediction result probability vector to obtain a final local prediction result probability vector for each of the multiple hybrid dual models in step S105.

As an implementation manner of the embodiment of the present invention, the step S105 may include:

b1: and adding the obtained local prediction result probability vectors aiming at each of the plurality of mixed double models to obtain a first local prediction vector.

For each of the multiple hybrid dual models, after obtaining multiple local predictor probability vectors for the hybrid dual model, the electronic device may add the obtained local predictor probability vectors to obtain a first local predictor vector.

For example, assume that for each hybrid dual model, the electronic device obtains 5 local predictor probability vectors, where the 5 local predictor probability vectors for a certain hybrid dual model are: x1 ═ 0.2,0.3,0.1,0.4, X2 ═ 0.3.0.2.0.0.5, X3 ═ 0,0,0.4,0.6, X4 ═ 0.1,0.3,0.2,0.4, and X5 ═ 0.4,0,0, 0.6. Then the first local predictor vector obtained by adding the 5 local predictor probability vectors is X ═ 1,0.8,0.7,2.5 for the hybrid dual-model.

B2: and normalizing the first local prediction vector to obtain a final local prediction result probability vector corresponding to the mixed dual model.

For each of the multiple mixed dual models, after obtaining the first local prediction vector, the electronic device may normalize the obtained first local prediction vector to obtain a final local prediction result probability vector corresponding to the mixed dual model.

For example, for a hybrid dual-model, the first local prediction vector is obtained as X ═ 0.5,0.8,1,1.3]Then, the first local prediction vector is normalized to obtain a final local prediction result probability vector R 'corresponding to the hybrid dual model'_patch＝[0.265,0.423,0.529,0.688]。

As another implementation manner of the embodiment of the present invention, the step S105 may also include,

c1: and determining the dimension corresponding to the element with the maximum value in each local prediction result probability vector as the label dimension aiming at each of the plurality of mixed double models.

For each of the multiple mixed dual models, after obtaining multiple local prediction result probability vectors of the mixed dual model, the electronic device may determine a dimension corresponding to an element with a largest value in each local prediction result probability vector as a tag dimension.

For example, assume that for each hybrid dual model, the electronic device obtains 5 local predictor probability vectors, where the 5 local predictor probability vectors for a certain hybrid dual model are: x1 ═ 0.2,0.3,0.1,0.4, X2 ═ 0.3.0.5.0.0.2, X3 ═ 0,0,0.4,0.6, X4 ═ 0.1,0.3,0.4,0.2, and X5 ═ 0.4,0,0, 0.6. The electronic device may determine that the label dimension of the local prediction result probability vector X1 is the fourth dimension, the label dimension of the local prediction result probability vector X2 is the second dimension, the label dimension of the local prediction result probability vector X3 is the fourth dimension, the label dimension of the local prediction result probability vector X4 is the third dimension, and the label dimension of the local prediction result probability vector X5 is the fourth dimension.

C2: and setting the numerical value of the element corresponding to the label dimension in each local prediction result probability vector as a first preset value, and setting the numerical values of the elements corresponding to the other dimensions as second preset values to obtain an initial local vector corresponding to each local prediction result probability vector.

After determining the label dimension of each local prediction result probability vector, the electronic device may set the value of the element corresponding to the label dimension in each local prediction result probability vector to a first preset value, and set the values of the elements corresponding to the remaining dimensions to a second preset value, to obtain an initial local vector corresponding to each local prediction result probability vector. The first preset value may be 1, and the second preset value may be 0, and of course, the first preset value and the second preset value may also be set to other values.

For example, it is assumed that, for each of the mixed dual models, the electronic device obtains 5 local prediction result probability vectors, a dimension of the local prediction result probability vector is 4, the first preset value is 1, the second preset value is 0, and for a certain mixed dual model, tag dimensions of the 5 local prediction result probability vectors corresponding to the mixed dual model, which are determined by the electronic device, are respectively: the label dimension of the local prediction result probability vector X1 is the fourth dimension, the label dimension of the local prediction result probability vector X2 is the second dimension, the label dimension of the local prediction result probability vector X3 is the fourth dimension, the label dimension of the local prediction result probability vector X4 is the third dimension, and the label dimension of the local prediction result probability vector X5 is the fourth dimension.

The electronic device obtains an initial local vector Y1 corresponding to the local predictor probability vector X1 as [0,0,0,1 ]; the initial local vector Y2 corresponding to the local predictor probability vector X2 is [0,1,0,0 ]; the initial local vector Y3 corresponding to the local predictor probability vector X3 is [0,0,0,1 ]; the initial local vector Y4 corresponding to the local predictor probability vector X4 is [0,0,1,0 ]; the initial local vector Y5 corresponding to the local predictor probability vector X5 is [0,0,1,0 ].

C3: and adding the obtained plurality of initial local vectors to obtain a second local prediction vector.

After obtaining the initial local vectors corresponding to each local predictor probability vector, the electronic device may add the obtained plurality of initial local vectors to obtain a second local predictor probability vector.

For example, suppose that for each of the mixed dual models, the electronic device can obtain 5 local prediction result probability vectors, and for a certain mixed dual model, the obtained 5 initial local vectors are: when Y1 is [0,0,0,1], Y2 is [0,1,0,0], Y3 is [0,0,0,1], Y4 is [0,0,1,0], Y5 is [0,0,1,0], the 5 initial local vectors are added to obtain a second local prediction result probability vector Y which is [0,1,1,3 ].

C4: and normalizing the second local prediction vector to obtain a final local prediction result probability vector corresponding to the mixed dual model.

After the second local prediction vector is obtained,the electronic device may normalize the obtained second local prediction vector to obtain a final local prediction result probability vector corresponding to the hybrid dual model. For example, for a hybrid dual-model, the obtained second local prediction vector is X ═ 0,1,4,2]Then, the second local prediction vector is normalized to obtain a final local prediction result probability vector R 'corresponding to the hybrid dual model'_patch＝[0,0.218,0.873,0.437]。

The following describes a manner of performing fusion calculation on the obtained final local prediction result probability vector and the final global prediction result probability vector according to a preset weight for each of the multiple hybrid dual models in step S107 to obtain a fused vector. As an implementation manner of the embodiment of the present invention, the step S107 may include:

for each of the plurality of hybrid dual models, a fused vector is calculated using the following formula:

R＝λR'_patch+(1-λ)R_global

wherein R is a fused vector, R'_patchFor the final local predictor probability vector, R_globalThe final global prediction result probability vector is obtained, lambda is the weight of the preset final local prediction result probability vector, and the value of lambda is [0,1]。

It should be noted that the weight λ of the preset final local prediction result probability vector may be set according to the precision requirement for classifying the pathological image based on the deep neural network in practical application, and the embodiment of the present invention does not limit the specific value of λ, and the value of λ may be [0,1] generally.

For example, assume that the number of hybrid bi-models is 3, hybrid bi-models a, b, c, respectively. λ is 0.3. For hybrid dual model a, final local predictor probability vector R'_patch＝[0,0,1,0]Final global predictor probability vector R_global＝[0,0,0,1]Then, the fused vector R is 0.3 × 0,0,1,0]+0.7*[0,0,0,1]＝[0,0,0.3,0.7](ii) a For hybrid dual model b, final local predictor probability vector R'_patch＝[0,1,0,0]Final global prediction resultProbability vector R_global＝[0,0,0,1]Then, the fused vector R is 0.3 × 0,1,0]+0.7*[0,0,0,1]＝[0,0.3,0,0.7](ii) a For hybrid dual model c, final local predictor probability vector R'_patch＝[0,0,0,1]Final global predictor probability vector R_global＝[0,0,0,1]Then, the fused vector R is 0.3 × 0,0,0,1]+0.7*[0,0,0,1]＝[0,0,0,1]。

The following describes a manner of determining the image type of the image to be measured in step S109 according to the prediction vector and the preset rule thereof. As an implementation manner of the embodiment of the present invention, step S109 may include:

d1: and determining the dimension corresponding to the element with the largest numerical value in the prediction vector as a target dimension.

After obtaining the prediction vector, the electronic device may determine a dimension of the element pair with the largest value in the prediction vector as a target dimension. For example, if the prediction vector is [0.2,0,0.8,0], the element with the largest value in the prediction vector is 0.8, the corresponding dimension is the third dimension, and the determined target dimension is the third dimension.

D2: and determining the image category corresponding to the target dimension according to the target dimension and the corresponding relation between the preset dimension and the preset image category.

The electronic device may preset a correspondence between the dimensions and the preset image categories, and store the correspondence. After the target dimension is determined, the electronic device may search for a preset image category corresponding to the target dimension in a corresponding relationship between the dimension and the preset image category, and then determine the image category corresponding to the target dimension.

For example, assuming that the preset image categories are A, B, C, D respectively, the dimension of the prediction vector is 4, and assuming that the corresponding relationship between the preset dimension and the preset image category is: the first dimension corresponds to a preset image category A, the second dimension corresponds to a preset image category B, the third dimension corresponds to a preset image category C, and the fourth dimension corresponds to a preset image category D. The electronic device executes step D2, and if the determined target dimension is the third dimension, the electronic device may determine that the image category corresponding to the target dimension is image category C.

D3: and taking the determined image category as the image category of the image to be detected.

After determining the image category corresponding to the target dimension, the electronic device may use the determined image category as the image category of the image to be detected. For example, assuming that the image category corresponding to the target dimension is image category C, the electronic device may determine that the image category of the image to be measured is image category C.

Corresponding to the pathological image classification method based on the deep neural network provided by the embodiment of the invention, the embodiment of the invention also provides a pathological image classification device based on the deep neural network.

As shown in fig. 3, a schematic structural diagram of a deep neural network-based pathological image classification apparatus according to an embodiment of the present invention is provided, where the apparatus includes:

an image to be tested acquisition module 310, configured to acquire an image to be tested;

the image cropping module 320 is configured to crop a local portion of the image to be detected according to a first preset size to obtain a plurality of cropped images to be detected, where the plurality of cropped images to be detected include all areas in the image to be detected.

The image reducing module 330 is configured to reduce the image to be detected according to a second preset size to obtain a global image to be detected, and reduce the plurality of cut images to be detected according to the second preset size to obtain a plurality of local images to be detected.

A detection vector obtaining module 340, configured to input, for each of the multiple hybrid dual models, multiple local images to be detected into a local network model for prediction, so as to obtain a local prediction result probability vector corresponding to each local image to be detected; and inputting the image to be detected into the global network model for prediction to obtain a global prediction result probability vector.

Wherein, each mixed dual model comprises a local network model and a global network model, and the local network model is as follows: the model training module is obtained by training a preset initial local depth neural network based on a local sample image and a corresponding class mark vector thereof, wherein the local sample image is as follows: after locally cutting a pre-acquired sample image according to a first preset size, reducing the image according to a second preset size to obtain an image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same, and the global network model is as follows: the model training module is obtained by training a preset initial global deep neural network based on a global sample image and a class mark vector corresponding to the global sample image, wherein the global sample image is as follows: and reducing the sample image according to a second preset size to obtain an image.

A first result processing module 350, configured to perform preset processing on each obtained local prediction result probability vector for each of the multiple hybrid dual models to obtain a final local prediction result probability vector.

And a second result processing module 360, configured to perform normalization processing on the obtained global prediction result probability vector for each of the multiple hybrid dual models, to obtain a final global prediction result probability vector.

And the fusion calculation module 370 is configured to perform fusion calculation on the obtained final local prediction result probability vector and the final global prediction result probability vector according to a preset weight for each of the multiple hybrid dual models to obtain a fused vector.

The predicted vector obtaining module 380 is configured to add a plurality of fused vectors corresponding to the plurality of mixed dual models to obtain a predicted vector.

The category determining module 390 is configured to determine an image category of the image to be detected according to the prediction vector and a preset rule.

As an implementation manner of the embodiment of the present invention, the model training module may include:

the initial network construction submodule is used for constructing an initial global deep neural network and an initial local deep neural network;

the image acquisition submodule is used for acquiring a first preset number of images from a pre-acquired sample image set as sample images and taking other images in the sample image set as verification images;

the vector determination submodule is used for determining a category mark vector corresponding to each sample image;

the global image obtaining submodule is used for reducing the sample image and the verification image according to a second preset size to obtain a global sample image and a global verification image;

the global network model training submodule is used for inputting the global sample image and the corresponding category label vector thereof into an initial global deep neural network for training to obtain an initial global network model;

the global network model testing submodule is used for inputting the global verification image into the initial global network model for prediction to obtain the initial global prediction accuracy;

the global network model judgment submodule is used for judging whether the first difference absolute value is not larger than a preset global model threshold value, wherein the first difference absolute value is as follows: if the absolute value of the difference value between the output result accuracy of the initial global network model and the initial global prediction accuracy is not greater than the absolute value, triggering the global network model to obtain a sub-module, and if not, triggering the global network model parameter adjusting sub-module;

the global network model obtaining submodule is used for stopping training and determining the initial global network model as a global network model;

the global network model parameter adjusting submodule is used for adjusting the parameters of the initial global deep neural network and triggering the global network model training submodule;

the image cutting submodule is used for carrying out local cutting according to a first preset size and respectively obtaining a second preset number of local sample cutting images and local verification cutting images aiming at each sample image and each verification image, wherein the class mark vectors corresponding to the local sample cutting images and the corresponding sample images are the same;

the local image obtaining submodule is used for reducing the obtained local sample cut image and the local verification cut image according to a second preset size to obtain a local sample image and a local verification image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same;

the local network model training submodule is used for inputting the local sample image and the corresponding class mark vector thereof into an initial local deep neural network for training to obtain an initial local network model;

the local network model testing submodule is used for inputting the local verification image into the initial local network model for prediction to obtain the initial local prediction accuracy;

the local network model judgment submodule is used for judging whether a second difference absolute value is not larger than a preset local model threshold value, wherein the second difference absolute value is as follows: if the absolute value of the difference value between the output result accuracy of the initial local network model and the initial local prediction accuracy is not greater than the absolute value, triggering a local network model obtaining submodule, and if the absolute value is greater than the absolute value, triggering a local network model parameter adjusting submodule;

the local network model obtaining submodule is used for stopping training and determining the initial local network model as a local network model;

and the local network model parameter adjusting submodule is used for adjusting the parameters of the initial local depth neural network and triggering the local network model training submodule.

As an implementation manner of the embodiment of the present invention, the vector determination sub-module may include:

the first probability determination unit is used for determining the probability that the image category of each sample image is a preset image category, wherein the preset image categories are multiple.

And the vector setting unit is used for setting a category vector, wherein the dimensionality of the category vector is the same as the number of the preset image categories.

And the second probability determining unit is used for determining the probability corresponding to the element corresponding to each dimensionality in the category vector according to the corresponding relation between the preset dimensionality and the preset image category and the determined probability.

And the vector obtaining unit is used for determining the numerical value of each element according to the probability corresponding to the element corresponding to each determined dimension to obtain the category label vector of each sample image.

As an implementation manner of the embodiment of the present invention, the first result processing module 350 may include:

and the first local prediction vector obtaining submodule is used for adding the obtained local prediction result probability vectors to obtain a first local prediction vector aiming at each of the plurality of mixed double models.

And the first final local prediction result probability vector obtaining submodule is used for normalizing the first local prediction vector to obtain a final local prediction result probability vector corresponding to the mixed dual model.

As an implementation manner of the embodiment of the present invention, the first result processing module 350 may further include:

and the label dimension determining submodule is used for determining the dimension corresponding to the element with the largest numerical value in each local prediction result probability vector as the label dimension aiming at each of the plurality of mixed double models. (ii) a

And the initial local vector obtaining submodule is used for setting the numerical value of the element corresponding to the label dimension in each local prediction result probability vector as a first preset value, and setting the numerical value of the element corresponding to the other dimensions as a second preset value to obtain the initial local vector corresponding to each local prediction result probability vector.

And the second local prediction vector obtaining submodule is used for adding the obtained plurality of initial local vectors to obtain a second local prediction vector.

The second final local prediction result probability vector obtaining sub-module 350D (not shown in fig. 3) is configured to normalize the second local prediction vector to obtain a final local prediction result probability vector corresponding to the hybrid dual model.

As an implementation manner of the embodiment of the present invention, the fusion calculation module 370 may be specifically configured to:

R＝λR'_patch+(1-λ)R_global

As an implementation manner of the embodiment of the present invention, the category determining module 390 may include:

and the target dimension determining submodule is used for determining the dimension corresponding to the element with the largest numerical value in the prediction vector as the target dimension.

And the target category determining submodule is used for determining the image category corresponding to the target dimension according to the target dimension and the corresponding relation between the preset dimension and the preset image category.

And the image category determining submodule is used for taking the determined image category as the image category of the image to be detected.

Corresponding to the pathological image classification method based on the deep neural network provided by the embodiment of the present invention, the embodiment of the present invention further provides an electronic device, as shown in fig. 4, including a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 complete mutual communication through the communication bus 404,

a memory 403 for storing a computer program;

the processor 401, when executing the program stored in the memory 403, is configured to implement the method steps of the pathological image classification method based on the deep neural network according to the embodiment of the present invention as described above:

specifically, the pathological image classification method based on the deep neural network includes:

acquiring an image to be detected;

locally cutting an image to be detected according to a first preset size to obtain a plurality of cut images to be detected, wherein the plurality of cut images to be detected comprise all areas in the image to be detected;

reducing the image to be detected according to a second preset size to obtain a global image to be detected, and reducing a plurality of cut images to be detected according to the second preset size to obtain a plurality of local images to be detected;

inputting a plurality of local images to be detected into a local network model for prediction aiming at each of a plurality of mixed double models to obtain a local prediction result probability vector corresponding to each local image to be detected; inputting the global image to be detected into a global network model for prediction to obtain a global prediction result probability vector; wherein, each mixed dual model comprises a local network model and a global network model, and the local network model is as follows: the method is obtained by training a preset initial local depth neural network based on a local sample image and a corresponding class mark vector thereof, wherein the local sample image is as follows: after locally cutting a pre-acquired sample image according to a first preset size, reducing the image according to a second preset size to obtain an image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same, and the global network model is as follows: the method is characterized by comprising the following steps of training a preset initial global deep neural network based on a global sample image and a class mark vector corresponding to the global sample image, wherein the global sample image is as follows: reducing the sample image according to a second preset size to obtain an image;

presetting each obtained local prediction result probability vector aiming at each of a plurality of mixed double models to obtain a final local prediction result probability vector;

adding a plurality of fused vectors corresponding to a plurality of mixed dual models to obtain a prediction vector;

It should be noted that other implementation manners of the pathological image classification method based on the deep neural network, which is implemented by the processor 401 executing the program stored in the memory 403, are the same as the embodiments of the pathological image classification method based on the deep neural network provided in the foregoing method embodiment section, and are not described herein again.

As can be seen from the above, in the embodiment of the present invention, when the electronic device trains each hybrid dual model, the global network model is trained using the global sample image, and the local network model is trained using the local sample image. Furthermore, when a plurality of mixed dual models are trained, the deep neural network can learn the global features and the local features of more image samples. Furthermore, the images to be detected and the local images to be detected corresponding to the images to be detected are predicted by utilizing the plurality of mixed models, so that the accuracy of the image category of the images to be detected determined according to the obtained plurality of detection results can be greatly improved.

The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

Corresponding to the pathological image classification method based on the deep neural network provided by the embodiment of the present invention, the embodiment of the present invention further provides a computer readable storage medium, which is a storage medium in a server, wherein a computer program is stored, and when being executed by a processor, the computer program implements the method steps of the pathological image classification method based on the deep neural network provided by the embodiment of the present invention.

acquiring an image to be detected;

It should be noted that other implementation manners of the pathological image classification method based on the deep neural network, which are implemented when the computer program is executed by the processor, are the same as the embodiments of the pathological image classification method based on the deep neural network provided in the foregoing method embodiment section, and are not described herein again.

As can be seen from the above, in the embodiment of the present invention, when the processor executes the computer program stored in the computer-readable storage medium, when each hybrid dual model is trained, the global network model is trained by using the global sample image, and the local network model is trained by using the local sample image. Furthermore, when a plurality of mixed dual models are trained, the deep neural network can learn the global features and the local features of more image samples. Furthermore, the images to be detected and the local images to be detected corresponding to the images to be detected are predicted by utilizing the plurality of mixed models, so that the accuracy of the image category of the images to be detected determined according to the obtained plurality of detection results can be greatly improved.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus embodiment, the electronic device embodiment and the computer-readable storage medium embodiment, since they are substantially similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A pathological image classification method based on a deep neural network is characterized by comprising the following steps:

acquiring an image to be detected;

inputting the local images to be measured into a local network model for prediction aiming at each of a plurality of mixed double models to obtain a local prediction result probability vector corresponding to each local image to be measured; inputting the global image to be detected into a global network model for prediction to obtain a global prediction result probability vector; each mixed dual-model comprises a local network model and a global network model, wherein the local network model is as follows: the method comprises the following steps of training a preset initial local depth neural network based on a local sample image and a corresponding class mark vector thereof, wherein the local sample image is obtained by: after locally cutting a pre-acquired sample image according to the first preset size, reducing the pre-acquired sample image according to the second preset size to obtain an image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same, and the global network model is as follows: the method comprises the following steps of training a preset initial global deep neural network based on a global sample image and a class mark vector corresponding to the global sample image, wherein the global sample image is obtained by: reducing the sample image according to the second preset size to obtain an image;

2. The method of claim 1, wherein the hybrid dual-model training mode comprises:

constructing an initial global deep neural network and an initial local deep neural network;

acquiring a first preset number of images from a pre-acquired sample image set as sample images, and taking other images in the sample image set as verification images;

determining a category label vector corresponding to each sample image;

reducing the sample image and the verification image according to the second preset size to obtain a global sample image and a global verification image;

inputting the global sample image and the corresponding category label vector thereof into the initial global deep neural network for training to obtain an initial global network model;

inputting the global verification image into the initial global network model for prediction to obtain initial global prediction accuracy;

judging whether a first difference absolute value is not larger than a preset global model threshold, wherein the first difference absolute value is as follows: the absolute value of the difference between the output result accuracy of the initial global network model and the initial global prediction accuracy;

stopping training and determining the initial global network model as the global network model if the first difference absolute value is not greater than the global model threshold;

if the absolute value of the first difference is larger than the threshold value of the global network model, adjusting the parameters of the initial global deep neural network, and executing the step of inputting the global sample image and the corresponding category label vector thereof into the initial global deep neural network for training to obtain an initial global network model;

performing local cropping according to the first preset size for each sample image and each verification image, and respectively obtaining a second preset number of local sample cropping images and local verification cropping images, wherein the class mark vectors corresponding to the local sample cropping images and the corresponding sample images are the same;

reducing the obtained local sample cut image and the local verification cut image according to the second preset size to obtain a local sample image and a local verification image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same;

inputting the local sample image and the corresponding class mark vector thereof into the initial local depth neural network for training to obtain an initial local network model;

inputting the local verification image into the initial local network model for prediction to obtain initial local prediction accuracy;

judging whether a second difference absolute value is not larger than a preset local model threshold, wherein the second difference absolute value is as follows: an absolute value of a difference between an output result accuracy of the initial local network model and the initial local prediction accuracy;

stopping training and determining the initial local network model as the local network model if the second difference absolute value is not greater than the local model threshold;

and if the second difference absolute value is larger than the local network model threshold, adjusting parameters of the initial local deep neural network, and executing the step of inputting the local sample image and the corresponding category label vector thereof into the initial local deep neural network for training to obtain an initial local network model.

3. The method of claim 2, wherein the step of determining the class label vector for each sample image comprises:

determining the probability that the image category of each sample image is a preset image category, wherein the number of the preset image categories is multiple;

setting category vectors, wherein the dimensionality of the category vectors is the same as the number of preset image categories;

determining the probability corresponding to the element corresponding to each dimension in the category vector according to the corresponding relation between the preset dimension and the preset image category and the determined probability;

and determining the numerical value of each element according to the determined probability corresponding to the element corresponding to each dimension to obtain the category label vector of each sample image.

4. The method according to claim 1, wherein the step of performing a predetermined process on the obtained local predictor probability vector to obtain a final local predictor probability vector for each of the plurality of hybrid dual models comprises:

adding the obtained local prediction result probability vectors to each of the plurality of mixed dual models to obtain a first local prediction vector;

and normalizing the first local prediction vector to obtain a final local prediction result probability vector corresponding to the mixed dual model.

5. The method according to claim 1, wherein the step of performing a predetermined process on each obtained local predictor probability vector to obtain a final local predictor probability vector for each of the plurality of hybrid dual models comprises:

determining a dimension corresponding to the element with the largest numerical value in each local prediction result probability vector as a label dimension for each of a plurality of mixed double models;

setting the numerical value of the element corresponding to the label dimension in each local prediction result probability vector as a first preset value, and setting the numerical values of the elements corresponding to the other dimensions as second preset values to obtain an initial local vector corresponding to each local prediction result probability vector;

adding the obtained plurality of initial local vectors to obtain a second local prediction vector;

and normalizing the second local prediction vector to obtain a final local prediction result probability vector corresponding to the mixed dual model.

6. The method according to claim 1, wherein the step of performing fusion calculation on the obtained final local predictor probability vector and the final global predictor probability vector according to a preset weight to obtain a fused vector for each of the plurality of hybrid dual models comprises:

R＝λR'_patch+(1-λ)R_global

wherein R is a fused vector, R'_patchFor the final local predictor probability vector, R_globalIn order to obtain the final global predictor probability vector,λ is the weight of the probability vector of the final local prediction result, and the value of λ is [0, 1%]。

7. The method according to any one of claims 1-6, wherein the step of determining the image type of the image to be tested according to the prediction vector and a preset rule comprises:

determining the dimension corresponding to the element with the largest numerical value in the prediction vector as a target dimension;

determining an image category corresponding to the target dimension according to the target dimension and the corresponding relation between the preset dimension and a preset image category;

and taking the determined image category as the image category of the image to be detected.

8. A pathological image classification device based on a deep neural network, the device comprising:

the detection vector acquisition module is used for inputting the local images to be detected into a local network model for prediction aiming at each of a plurality of mixed double models to obtain a local prediction result probability vector corresponding to each local image to be detected; inputting the global image to be detected into a global network model for prediction to obtain a global prediction result probability vector; each mixed dual-model comprises a local network model and a global network model, wherein the local network model is as follows: the model training module is obtained by training a preset initial local depth neural network based on a local sample image and a corresponding class mark vector thereof, wherein the local sample image is as follows: after locally cutting a pre-acquired sample image according to the first preset size, reducing the pre-acquired sample image according to the second preset size to obtain an image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same, and the global network model is as follows: the model training module is obtained by training a preset initial global deep neural network based on a global sample image and a class mark vector corresponding to the global sample image, wherein the global sample image is as follows: reducing the sample image according to the second preset size to obtain an image;

9. The apparatus of claim 8, wherein the model training module comprises:

the image acquisition sub-module is used for acquiring a first preset number of images from a pre-acquired sample image set as sample images and taking other images in the sample image set as verification images;

the global image obtaining submodule is used for reducing the sample image and the verification image according to the second preset size to obtain a global sample image and a global verification image;

the global network model training submodule is used for inputting the global sample image and the corresponding category label vector thereof into the initial global deep neural network for training to obtain an initial global network model;

the global network model testing sub-module is used for inputting the global verification image into the initial global network model for prediction to obtain initial global prediction accuracy;

the global network model judgment submodule is used for judging whether a first difference absolute value is not larger than a preset global model threshold, wherein the first difference absolute value is as follows: if the absolute value of the difference value between the output result accuracy of the initial global network model and the initial global prediction accuracy is not greater than the absolute value, triggering a global network model obtaining sub-module, and if not, triggering a global network model parameter adjusting sub-module;

the global network model obtaining submodule is used for stopping training and determining the initial global network model as the global network model;

the image cropping submodule is used for performing local cropping according to the first preset size aiming at each sample image and each verification image and respectively obtaining a second preset number of local sample cropping images and local verification cropping images, wherein the class mark vectors corresponding to the local sample cropping images and the corresponding sample images are the same;

the local image obtaining submodule is used for reducing the obtained local sample cut image and the local verification cut image according to the second preset size to obtain a local sample image and a local verification image, wherein the class mark vectors corresponding to the local sample image and the corresponding sample image are the same;

the local network model training submodule is used for inputting the local sample image and the corresponding class mark vector thereof into the initial local deep neural network for training to obtain an initial local network model;

the local network model testing sub-module is used for inputting the local verification image into the initial local network model for prediction to obtain initial local prediction accuracy;

the local network model judgment submodule is used for judging whether a second difference absolute value is not larger than a preset local model threshold, wherein the second difference absolute value is as follows: if the absolute value of the difference value between the output result accuracy of the initial local network model and the initial local prediction accuracy is not greater than the absolute value, triggering a local network model obtaining sub-module, and if the absolute value is greater than the absolute value, triggering a local network model parameter adjusting sub-module;

the local network model obtaining submodule is used for stopping training and determining the initial local network model as the local network model;

the local network model parameter adjusting submodule is used for adjusting the parameters of the initial local deep neural network and triggering the local network model training submodule.

10. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.