CN112699950A

CN112699950A - Medical image classification method, image classification network processing method, device and equipment

Info

Publication number: CN112699950A
Application number: CN202110012705.5A
Authority: CN
Inventors: 边成
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-01-06
Filing date: 2021-01-06
Publication date: 2021-04-23
Anticipated expiration: 2041-01-06
Also published as: CN112699950B

Abstract

The present application relates to artificial intelligence technology, and in particular, to a medical image classification method, apparatus, computer device, and storage medium. The method comprises the following steps: slicing the medical image to obtain an image slice sequence; extracting image features of an object in each image slice in the image slice sequence; sequentially determining the continuous probability of the target object in each image slice according to the image features respectively extracted from each image slice; determining the communication rate of the target object in the image slice sequence based on the accumulated sum of difference values between continuous probabilities corresponding to each adjacent image slice in the image slice sequence and the total slice amount of the image slices in the image slice sequence; and judging the classification category of the medical image according to the communication rate. By adopting the method, the influence on the accuracy of medical image classification caused by smaller target objects can be avoided.

Description

Medical image classification method, image classification network processing method, device and equipment

Technical Field

The present application relates to the field of artificial intelligence technologies, and in particular, to a medical image classification method, an image classification network processing apparatus, and a medical image classification network processing device.

Background

A medical image is an image of internal tissue obtained non-invasively from the human body or a part of the human body for medical treatment or medical research. Performing a classification process on the medical image to obtain the medical image classification category may provide effective medical assistance for medical workers.

In a conventional medical image classification scheme, a convolutional neural network is generally used to segment a target object in a medical image, and then the medical image is classified according to the segmented target object. However, when the target object in the medical image is small, the medical image segmentation may fail or be inaccurate, thereby affecting the accuracy of medical image classification.

Disclosure of Invention

In view of the above, it is necessary to provide a medical image classification method, an image classification network processing device, and a medical image classification network processing device, which can avoid the influence on the accuracy of medical image classification due to a small target object.

A method of medical image classification, the method comprising:

slicing the medical image to obtain an image slice sequence;

extracting image features of an object in each image slice in the image slice sequence;

sequentially determining the continuous probability of the target object in each image slice according to the image features respectively extracted from each image slice;

determining the communication rate of the target object in the image slice sequence based on the accumulated sum of difference values between continuous probabilities corresponding to each adjacent image slice in the image slice sequence and the total slice amount of the image slices in the image slice sequence;

and judging the classification category of the medical image according to the communication rate.

In one embodiment, before the slicing process is performed on the medical image sample, the method further comprises:

when a medical image sample is obtained, normalizing the medical image sample based on the mean and variance of the medical image sample;

carrying out image overturning on the processed medical image sample to obtain an overturned medical image sample;

the slicing processing of the medical image sample to obtain a sample slice sequence includes:

and slicing the turned medical image sample to obtain a sample slice sequence.

In one embodiment, the extracting, by a first feature extractor in the first image classification network, the first training image feature of the object in each group of sample slices is preceded, and the method further includes:

sequentially inputting sample slices in the sample slice sequence into the second image classification network;

extracting, by a second feature extractor in the second image classification network, a second training image feature of the target object in the input sample slice;

performing target object continuity calculation on the extracted second training image features through a second classifier in the second image classification network to obtain a second target continuity probability;

calculating a second loss value between the second target continuity probability and a reference continuity rate;

adjusting the parameters of the second feature extractor and the parameters of the second classifier according to the second loss value;

and taking the adjusted second feature extractor as a first feature extractor in the first image classification network.

A medical image classification apparatus, the apparatus comprising:

the processing module is used for carrying out slice processing on the medical image to obtain an image slice sequence;

the extraction module is used for extracting image features of the object in the image slices from the image slices in the image slice sequence;

the determining module is used for sequentially determining the continuous probability of the target object in each image slice according to the image features extracted from each image slice; determining the communication rate of the target object in the image slice sequence based on the accumulated sum of difference values between continuous probabilities corresponding to each adjacent image slice in the image slice sequence and the total slice amount of the image slices in the image slice sequence;

and the judging module is used for judging the classification category of the medical image according to the connectivity.

A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

slicing the medical image to obtain an image slice sequence;

A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:

slicing the medical image to obtain an image slice sequence;

According to the medical image classification method, the medical image classification device, the computer equipment and the storage medium, the medical image is sliced to obtain the image slice sequence, and the image features of the target object in each image slice of the image slice sequence are extracted, so that the calculated amount of the extracted image features is reduced, and the feature extraction speed is increased. The continuous probability of the target object in each image slice is sequentially determined according to the extracted image features, and the connectivity in the image slice sequence of the target object is determined based on the accumulated sum of difference values between the continuous probabilities of adjacent image slices and the total number of the slices, so that the information between different image slices is obtained, and the relevance between different image slices is considered, therefore, the classification category to which the medical image belongs is judged according to the connectivity, and the classification of the medical image can be more accurate. In addition, because the target object does not need to be segmented in each image slice, even if the target object is small, the classification category of the medical image can be judged based on the connectivity of the target object in the image slice sequence, and the accuracy of medical image classification can be improved.

An image classification network processing method, the method comprising:

slicing a medical image sample to obtain a sample slice sequence;

inputting a grouping of sample slices in the sequence of sample slices into a first image classification network;

extracting first training image features of the target object in each sample slice of each group through a first feature extractor in a first image classification network;

performing target object continuity calculation on the extracted first training image features through a first classifier in the first image classification network to obtain a first target continuity probability; and determining a target communication rate of the target object in the sample slice sequence based on the accumulated sum of the difference values between the first target continuous probabilities and the total slice amount of the sample slice sequence;

calculating a first loss value between the target connectivity rate and a reference connectivity rate;

and adjusting the parameters of the first feature extractor and the parameters of the first classifier according to the first loss value.

In one embodiment thereof, the method further comprises:

and respectively initializing parameters in the first image classification network and the second image classification network based on the normal distribution of the target mean and the target variance.

and slicing the turned medical image sample to obtain a sample slice sequence.

In one embodiment, before the inputting the group of sample slices in the sequence of sample slices into the first image classification network, the method further comprises:

An image classification network processing apparatus, the apparatus comprising:

the processing module is used for carrying out slicing processing on the medical image sample to obtain a sample slice sequence;

an input module for inputting a group of sample slices in the sequence of sample slices into a first image classification network;

the extraction module is used for extracting first training image features of the target object in each sample slice of each group through a first feature extractor in the first image classification network;

the determining module is used for carrying out target object continuity calculation on the extracted first training image features through a first classifier in the first image classification network to obtain a first target continuity probability; and determining a target communication rate of the target object in the sample slice sequence based on the accumulated sum of the difference values between the first target continuous probabilities and the total slice amount of the sample slice sequence;

the calculation module is used for calculating a first loss value between the target connectivity and a reference connectivity;

and the adjusting module is used for adjusting the parameters of the first feature extractor and the parameters of the first classifier according to the first loss value.

slicing a medical image sample to obtain a sample slice sequence;

According to the image classification network processing method, the image classification network processing device, the computer equipment and the storage medium, the sample slice sequence is obtained by slicing the medical image sample, and the first feature extractor in the first image classification network extracts the first training image features of the target object in each image slice of the sample slice sequence, so that the calculated amount of extracting the first training image features is reduced, and the feature extraction speed is increased. A first classifier in the first image classification network sequentially determines first target continuous probability of a target object in each sample slice according to extracted first training image features, and determines target communication rate in a target object sample slice sequence based on the accumulated sum of difference values between the first target continuous probabilities of adjacent sample slices and the total slice amount of the sample slice sequence, so that the first image classification network learns the correlation between different sample slices, the first image classification network can extract correlation information between image slices of different medical images, the performance of the first image classification network is improved, and the accuracy of medical image classification is improved.

Drawings

FIG. 1 is a diagram of an embodiment of an application environment of a medical image classification method;

FIG. 2 is a flow diagram of a medical image classification method in one embodiment;

FIG. 3 is a diagram illustrating a slicing process for a medical image in one embodiment;

FIG. 4 is a diagram illustrating an embodiment of feature extraction and classification of image slices by a feature extractor and a classifier, respectively;

FIG. 5 is a flowchart illustrating the steps of marking and displaying the location of the break in the target object in the medical image according to one embodiment;

FIG. 6 is a schematic representation of an embodiment of an image slice with no and no breaks;

FIG. 7 is a flowchart illustrating the risk-alerting step in one embodiment;

FIG. 8 is a flow diagram that illustrates the processing steps performed on an image classification network in one embodiment;

FIG. 9 is a diagram illustrating model training for two different image classification networks in two stages, respectively, according to an embodiment;

FIG. 10 is a flowchart illustrating a method for image classification network processing according to one embodiment;

FIG. 11 is a block diagram showing the structure of a medical image classification apparatus according to an embodiment;

FIG. 12 is a block diagram showing the construction of a medical image classification apparatus according to another embodiment;

FIG. 13 is a block diagram showing the structure of an image classification network processing apparatus according to an embodiment;

FIG. 14 is a block diagram showing the structure of an image classification network processing apparatus according to another embodiment;

FIG. 15 is a diagram showing an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein.

Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision technology (CV) is a science for researching how to make a machine "see", and further refers to that a camera and a Computer are used to replace human eyes to perform machine Vision such as identification, tracking and measurement on a target, and further image processing is performed, so that the Computer processing becomes an image more suitable for human eyes to observe or is transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. Computer vision technologies generally include technologies such as image segmentation, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning, map construction, and the like, and also include common biometric technologies such as face recognition, fingerprint recognition, and the like.

Machine Learning (ML) is a multi-domain cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, formal learning, metric learning, and the like.

The scheme provided by the embodiment of the application relates to the computer vision technology of artificial intelligence, the machine learning technology and the like, and is specifically explained by the following embodiment:

the medical image classification method provided by the application can be applied to the application environment shown in fig. 1. In the application environment, a terminal 102, a server 104, a database 106, and a medical device 108 are included. The medical image classification method may be applied to at least one of the terminal 102, the server 104, and the medical device 108. Taking the example that the medical image classification method is applied to the server 104, the server 104 may directly slice the medical image when acquiring the medical image from the database, where the medical image is acquired by the medical device; or, when receiving a classification instruction for a medical image from the terminal 102, acquiring a medical image corresponding to the classification instruction, and then performing slicing processing on the medical image, thereby obtaining an image slice sequence; extracting image features of the target object in the image slices from each image slice in the image slice sequence; sequentially determining the continuous probability of the target object in each image slice according to the image features respectively extracted from each image slice; determining the communication rate of the target object in the image slice sequence based on the accumulated sum of the difference values between the continuous probabilities corresponding to each adjacent image slice in the image slice sequence and the total slice amount of the image slices in the image slice sequence; the classification category to which the medical image belongs is determined according to the connectivity rate, and then the classification category is displayed on the terminal 102.

The terminal 102 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like.

The server 104 may be a separate physical server or may be integrated into a medical device for medical image classification; in addition, the server 104 may also be a server cluster composed of a plurality of physical servers, and may be a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a web service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.

The database 106 may be a relational, non-relational, and key-value database for storing medical images acquired by the medical device 108.

The medical device 108 may be a device for medical treatment or medical research, including but not limited to an OCT (Optical Coherence Tomography), other devices for acquiring images of the sub-ocular region, a CT (Computed Tomography) detector, an MRI (Magnetic Resonance Imaging) detector, and the like. After the medical device 108 acquires the medical image, the acquired medical image may be stored in the database 106, or the medical image may be directly transmitted to the server 104, and in the actual application process, the acquired medical image may also be transmitted to the terminal 102 for display.

The terminal 102, the server 104, the database 106 and the medical device may be connected through bluetooth, USB (Universal Serial Bus), or network, which is not limited herein.

In one embodiment, as shown in fig. 2, a medical image classification method is provided, which can be applied to at least one of the terminal 102, the server 104 and the medical device 108 in fig. 1, and is exemplified by the application of the method to the server 104 in fig. 1, and includes the following steps:

s202, slicing the medical image to obtain an image slice sequence.

The medical image may be an image obtained by non-invasively capturing a human body or a part of a human body by medical equipment, or an image obtained by capturing an image of another object by medical equipment, where the other object may include various animals, plants, microorganisms, and the like. For example, the medical image may be an OCT image obtained by performing optical coherence tomography on an eye with an OCT apparatus.

In one embodiment, the server may retrieve the medical image from the database according to the image classification instructions, or the server may retrieve the medical image directly from the medical device when the medical device acquires the medical image.

After acquiring the medical image, the medical device may store the medical image in a database, so that when receiving an image classification instruction of the terminal for the medical image, the medical device acquires the medical image from the database for image classification, where the image classification instruction may be triggered by a classification button on an image operation page of the terminal, as shown in fig. 1. In addition, the medical image can also be directly sent to the server, so that the server classifies the medical image in real time.

In one embodiment, S202 may specifically include: the server determines the width of the slice and the step length of the gradual movement during the slice processing; gradually cutting out image slices with slice widths in the medical image according to step lengths to obtain an image slice sequence; or the server determines a region of interest in the medical image and cuts out image slices of slice width step by step in the region of interest according to step length to obtain an image slice sequence. The medical image is subjected to slice processing, so that the image characteristics of the target object can be extracted in each image slice, the calculation amount for extracting the image characteristics is reduced, and the characteristic extraction speed is increased.

When the slicing processing is performed, the server may determine a segmentation starting point in the medical image, and begin to segment the medical image with the segmentation starting point to obtain an image slice having a width consistent with the determined slice width; then, the slice sequence is obtained by moving the slice sequence step by step from the starting point of the segmentation according to the determined step size and segmenting the image slice in the medical image according to the determined slice width after each movement.

Or, when the slicing processing is performed, the image recognition is performed on the edge area of the medical image, and when the edge area of the medical image is determined to be an invalid area, the server may cut the edge area from the medical image, so as to obtain an area of interest; determining a segmentation starting point in the region of interest, and beginning to segment the region of interest by using the segmentation starting point to obtain an image slice with the width consistent with the determined slice width; then, the image slice sequence is obtained by moving step by step in a determined step from the starting point of the segmentation, and after each movement, segmenting an image slice in the region of interest which is consistent with the determined slice width.

For example, as shown in fig. 3, when obtaining an OCT image of an eye position, the server may directly move step by step in a specific step and perform a slicing process on the OCT image while moving, thereby obtaining an image slice sequence. Or the server cuts the invalid region of the OCT image to obtain the region of interest; then, the region of interest is moved step by step in a specific step size, and the region of interest is sliced while moving, so that an image slice sequence is obtained.

In one embodiment, before performing the slicing processing on the medical image, the server may further perform image enhancement processing on the medical image, and then perform the slicing processing on the medical image after the image enhancement processing to obtain the image slice sequence. In addition, after the image enhancement processing, the server can also perform normalization processing on the medical image after the image enhancement processing, and then perform slicing processing on the medical image after the normalization processing to obtain an image slice sequence.

When image enhancement is performed, the following method can be adopted to perform image enhancement on the medical image: the server adjusts the brightness of the medical image, and/or adjusts the chrominance of the medical image, and/or adjusts the sharpness of the medical image, etc.

When normalization processing is performed, the normalization processing can be performed on the medical image after the image enhancement processing in the following way: the server calculates the mean value and the variance of the medical image after the image enhancement processing, and normalizes the medical image after the image enhancement processing according to the calculated mean value and variance. For example, assuming that the matrix of the medical image after the image enhancement processing is L, the terminal calculates the sum and variance of the matrix L as u and δ, respectively, and the result after the normalization processing is L' ═ L-u)/δ.

And S204, extracting the image characteristics of the target object in the image slices from each image slice in the image slice sequence.

The object may be an object that affects the classification of medical images, for example, for medical images of eye positions (such as OCT images of eyes), the object may be an Ellipsoid Zone (EZ) and/or an External Limiting Membrane (ELM), as shown in fig. 3, in which the Ellipsoid Zone is shown by a thinner curve in the figure and the External Limiting Membrane is shown by a thicker gray curve in the figure. It should be noted that the above examples of the object are only examples, and include, but are not limited to, the above objects affecting the classification of medical images.

In one embodiment, the server may perform feature extraction on each image slice in the image slice sequence through a first feature extractor of the first image classification network, to obtain an image feature of the object in each image slice. The first feature extractor may be any one of ResNet101, MobileNet, and DenseNet, or may be another neural network for extracting image features.

The server inputs the image slice sequence into a first image classification network, and calculates each image slice in the input image slice sequence through each network layer in the first feature extractor to obtain the image feature of the target object in each image slice. The network layer of the first feature extractor comprises a convolution layer, a pooling layer, a normalization layer + ReLu, a full connection layer and a Softmax layer. When the image classification network is input, the server may input the image slices in the image slice sequence in the order of the slices, may input the image slices in the image slice sequence in batches, that is, each time a group of image slices are input, and may input the image slices in the image slice sequence at one time.

In one embodiment, S204 may specifically include: the server groups the image slices in the image slice sequence according to the slice sequence to obtain at least two groups of image slices; and respectively carrying out feature extraction on the image slices of each group through a first feature extractor in the first image classification network to obtain the image features of the target object in the image slices of each group.

Wherein grouping may refer to dividing at least two adjacent image slices into one group. The slice order includes an order when the medical image is subjected to slice processing. Therefore, the grouping method is as follows: the server sorts the image slices in the image slice sequence according to the slice sequence; determining the number of slices in the group; and grouping the image slices in the ordered image slice sequence according to the number of slices in the group.

The server firstly groups the image slices in the image slice sequence according to the slice sequence to obtain at least two groups of image slices, then sequentially inputs the image slices of each group into a first image classification network, and calculates the image slices of the input group through each network layer in a first feature extractor to obtain the image features of the target object in the image slices of each group.

For example, as shown in fig. 4, the server sequentially slices the OCT image of the eye position to obtain three image slices, then respectively inputs the obtained image slices into the feature extractor of the first image classification network, performs convolution processing on two convolution layers in the feature extractor, and then sequentially performs normalization processing and ReLu processing on the result of the convolution processing through the normalization layer + ReLu to obtain a first image feature; secondly, pooling the first image characteristic through a pooling layer, performing convolution processing on a result of the pooling processing through two convolution layers, and performing normalization processing and ReLu processing on the result of the convolution processing through a normalization layer and ReLu to obtain a second image characteristic; continuously performing pooling processing on the second image characteristic through a pooling layer, performing convolution processing on a result of the pooling processing through two convolution layers, and performing normalization processing and ReLu processing on the result of the convolution processing through a normalization layer and ReLu to obtain a third image characteristic; finally, the third image feature is subjected to pooling processing by the pooling layer, and then the result of the pooling processing is processed by the full-connected layer, so as to obtain image features of the target object in the three image slices, such as feature 1, feature 2 and feature 3 in the figure. And so on, the server can obtain the image characteristics of the object in each image slice.

And S206, sequentially determining the continuous probability of the target object in each image slice according to the image features respectively extracted from each image slice.

The continuous probability may refer to the probability that the object is continuous in each image slice, and if a break (or a breakpoint) occurs in one or more image slices of the object, the object is not continuous in the one or more image slices; if the object does not break (or break point) in one or more of the image slices, the object is continuous in the one or more image slices.

In one embodiment, the server may perform anomaly identification on the image features respectively extracted from each image slice, for example, identify whether the extracted image features are broken or not, and obtain an identification result; then, the continuous probability of the object in each image slice is determined according to the identification result.

Specifically, the server performs anomaly identification on image features respectively extracted from each image slice through a first classifier in a first image classification network to obtain an identification result indicating whether a target object is broken in each image slice; determining the continuous probability of the target object in each image slice based on the identification result; or determining the fracture probability of the target object in each image slice according to the identification result, and obtaining the continuous probability according to the fracture probability. The first classifier may be implemented as an LSTM (Long Short-Term Memory) network or a recurrent neural network.

The first classifier can perform anomaly identification on the image features respectively extracted from each image slice to determine whether the extracted image features are fractured or not, and obtain an identification result, wherein if the image features of a certain image slice are fractured, a fracture mark related to the image slice is obtained, or a fracture probability related to the image slice is obtained; if the image features of an image slice are continuous, a continuous marker or a continuous probability is obtained for the image slice. Thus, the continuous probability of the target object in each image slice can be determined according to the recognition result.

S208, determining the communication rate of the target object in the image slice sequence based on the accumulated sum of the difference values between the continuous probabilities corresponding to each adjacent image slice in the image slice sequence and the total slice amount of the image slices in the image slice sequence.

Wherein, the difference value may refer to an absolute value of a difference between consecutive probabilities respectively corresponding to two adjacent image slices. For example, if the consecutive probability of the target object in the image slice a1 is p1 and the consecutive probability of the target object in the image slice a2 is p2 in the two adjacent image slices a1 and a2, the difference value is p ═ p1-p2 |.

Specifically, for each adjacent image slice in the image slice sequence, the server performs difference calculation on the continuous probability corresponding to each adjacent image slice to obtain a difference value between the continuous probabilities corresponding to each adjacent image slice; then, the absolute value of the difference value is taken to obtain a difference value, and the difference values are summed to obtain an accumulated sum. The server determines a connectivity rate of the object in the sequence of image slices based on the accumulated sum and a total number of slices of the image slices in the sequence of image slices.

For example, if the image slices in the image slice sequence are a (1), …, a (n), and the image slices a (1), …, a (n) respectively correspond to consecutive probabilities p (1), …, p (n), the general formula of the difference value between the consecutive probabilities corresponding to adjacent image slices is Diff¹(i) P (i) -P (i-1) |, so that the cumulative sum is P ═ Σ_iDiff¹(i) In that respect And when the accumulated sum and the slice number of the image slice sequence are obtained, inputting the accumulated sum and the slice number into a communication rate calculation formula, and obtaining the communication rate of the target object in the image slice sequence through the communication rate calculation formula. Wherein, the communication ratio calculation formula is as follows:

wherein n represents the number of slices of the sequence of image slices, i.e. the total number of image slices in the sequence of image slices; sigma_iDiff¹(i) Representing a cumulative sum, n being a positive integer greater than 1, and i being a positive integer less than or equal to n.

In one embodiment, the communication rate is a first order communication rate; the method further comprises the following steps: the server determines a difference value between adjacent connectivity rates in the first-order connectivity rate; summing the absolute values of the difference values to obtain an accumulated value; and determining the second-order communication rate of the object in the image slice sequence according to the accumulated value and the number of the first-order communication rates.

The server can calculate a second-order communication rate on the basis of the first-order communication rate so as to judge the classification category to which the medical image belongs according to the second-order communication rate. The second-order communication rate may be calculated in the following manner: calculating the difference of two adjacent communication rates, then summing the absolute values of the difference values, and taking the ratio of the accumulated value obtained by summation to the number of the communication rates as a second-order communication rate; or the difference between one and this ratio is taken as the second order rate of communication.

And S210, judging the classification type of the medical image according to the connectivity.

Wherein, the classification category may refer to: the type of the medical image to which the object belongs in a continuous or broken state presented in the medical image, for example, the type of abnormality of the object or the type of normality of the object. For example, when a break occurs in one or a plurality of consecutive image slices of a medical image of a target object, the medical image is of a type in which an abnormality occurs in the target object.

Different communication rate intervals can be preset according to actual conditions, and the different communication rate intervals can correspond to different classification categories, such as a first category, a second category, a third category and the like. When the connectivity is in a first interval, the server judges that the medical image belongs to a first category; when the connectivity is in the second interval, the server determines that the medical image belongs to the second category, and so on.

In practical application, the communication rate intervals may be divided according to whether the target object is abnormal or not and the severity of the abnormal condition, so that different communication rate intervals correspond to different types, for example: normal type, mild abnormal type, moderate abnormal type, and severe abnormal type. For example, when a target object is broken in one image slice in a medical image, the medical image belongs to a mild anomaly type; when a fracture occurs in m image slices of a medical image, the medical image belongs to a moderate abnormal type, wherein m is smaller than or equal to a threshold value T; when a fracture occurs in q image slices of a medical image, the medical image belongs to a severe anomaly type, wherein q is larger than a threshold value T. It is noted that the larger the number of image slices in which a fracture occurs, the smaller the connectivity.

In one embodiment, when the number of the objects is not less than two, weighted summation is carried out on the connectivity of each object, and the classification category to which the medical image belongs is judged according to the connectivity sum value obtained by the weighted summation. Or the server determines a connectivity rate interval where the connectivity rate of each target object is located to obtain a determination result, and then searches the classification table according to the determination result to obtain a classification category matched with the determination result. It should be noted that, the connectivity of different targets falls in different connectivity intervals, and the corresponding classification categories may be different.

In one embodiment, when calculating the second-order connectivity rate, the server may determine the classification category to which the medical image belongs according to the second-order connectivity rate. When a plurality of image slices are input into the image classification network each time, the relevance among a plurality of different image slices can be more favorably obtained through the calculation of the second-order communication rate, so that the classification category to which the medical image belongs is judged according to the second-order communication rate, and the classification of the medical image can be more accurate.

As an example, the medical image is an OCT image of an eye position, and the target is an ellipsoid zone and a retina outer boundary membrane as an example for explanation: firstly, a server acquires an OCT image, and slices the OCT image to obtain an OCT slice sequence, wherein the OCT slice sequence comprises OCT slices a (1), a (2), … and a (n); then, extracting image features of the ellipsoidal zone and the retina outer limiting membrane from the OCT slices a (1), a (2), …, a (n), respectively, and sequentially determining continuous probabilities of the ellipsoidal zone and the retina outer limiting membrane in each OCT slice a (1), a (2), …, a (n), respectively, according to the extracted image features; then, the absolute value of the difference between the continuous probabilities of adjacent OCT slices in OCT slices a (1), a (2), …, a (n) is calculated, the connectivity of the ellipsoid zone and the retina outer limiting membrane in the OCT slice sequence is determined according to the accumulated sum of the absolute values of the difference and the number of the OCT slices, and if the connectivity of the ellipsoid zone and the retina outer limiting membrane is in a connectivity interval with a smaller connectivity, the ellipsoid zone and the retina outer limiting membrane are determined to be abnormal.

In the above embodiment, the image slice sequence is obtained by performing slice processing on the medical image, and the image features of the target object in each image slice of the image slice sequence are extracted, so that the amount of calculation for extracting the image features is reduced, and the rate of feature extraction is increased. The continuous probability of the target object in each image slice is sequentially determined according to the extracted image features, and the connectivity in the image slice sequence of the target object is determined based on the accumulated sum of difference values between the continuous probabilities of adjacent image slices and the total number of the slices, so that the information between different image slices is obtained, and the relevance between different image slices is considered, therefore, the classification category to which the medical image belongs is judged according to the connectivity, and the classification of the medical image can be more accurate. In addition, because the target object does not need to be segmented in each image slice, even if the target object is small, the classification category of the medical image can be judged based on the connectivity of the target object in the image slice sequence, and the accuracy of medical image classification can be improved.

In one embodiment, as shown in fig. 5, the method may further include:

s502, when the target object is determined to break in the target image slice of the image slice sequence according to the recognition result, the breaking position of the target object in the target image slice is determined.

In which the object is broken in the target image slice, indicating that the object has an abnormality in the target image slice, for example, an ellipsoidal band and/or a defect in the outer retina at a certain position may affect the vision of the user. The fracture location may be determined from a pixel coordinate system. The fracture position of the object in the target image slice comprises: the object may be present only on one, both or the middle of the target image slice, or may not be present in the target image slice. As shown in fig. 6, (a) in the figure shows a case where the object is not broken in the image slice, and (b) to (f) in the figure show a case where the object is broken in the image slice.

For example, a pixel coordinate system is established in the target image slice, and the coordinates of the fracture pixel points in the pixel coordinate system are determined, so that the fracture positions of the target object in the target image slice can be obtained.

And S504, marking a position area matched with the fracture position in the medical image.

In one embodiment, the server finds a location area in the medical image that matches the determined fracture location, i.e. finds the fracture location in the medical image; the located location area is then marked with a different identifier than the medical image, e.g. with a box of a different color than the medical image.

S506, after the classification category to which the medical image belongs is judged, the marked medical image and the classification category are displayed.

The server may transmit the tagged medical images and classification categories to the terminal, and display the tagged medical images and classification categories through the terminal.

In the embodiment, when the target object is fractured in the target image slice, the fracture position is found, the fracture position is marked in the medical image, and the classification type of the medical image and the marked medical image are displayed, so that the specific type can be obtained, the abnormal position can be visually and clearly seen, the worker is prevented from finding the abnormal position in the larger medical image repeatedly, and the abnormal position can be quickly located.

In one embodiment, as shown in fig. 7, the method may further include:

s702, when the target object is determined to have fracture in a plurality of target image slices of the image slice sequence according to the identification result, the continuity of the sequence numbers among the plurality of target image slices is detected.

Each image slice in the image slice sequence has a corresponding serial number, and the serial number may be a serial number obtained by numbering the image slices according to a slice sequence when the slice processing is performed. For example, when an OCT image of an eye position is sliced from left to right, each time an image slice is sliced, the image slices are numbered sequentially to obtain corresponding serial numbers.

S704, the number of the continuous slices in the target image slices is determined according to the continuity.

The consecutive slices are image slices having consecutive numbers. The number of slices may refer to: when the object breaks in the consecutive target image slices, the number of the consecutive target image slices is the slice number.

In one embodiment, S704 may specifically include: the server determines a continuous slice group in the target image slices according to the continuity; acquiring the number of slices in each continuous slice group; the number of slices comprises the number of slices in the group; and selecting the largest number of the group internal slices from the obtained number of the group internal slices.

When a plurality of continuous slice groups exist, the number of slices in each continuous slice group is counted to obtain the number of slices in each continuous slice group, and then the maximum number of slices in each continuous slice group is obtained.

S706, determining the ratio of the number of the slices to the total number of the slices.

In one embodiment, after determining the ratio of the number of slices to the total number of slices, the server may further use the ratio as an abnormal rate; or after the maximum number of slices of the continuous slice group is obtained, the server may calculate a ratio of the maximum number of slices in the group to the total number of slices, and use the ratio of the maximum number of slices in the group to the total number of slices as an abnormal rate, where the abnormal rate is the maximum abnormal rate.

And S708, after the classification type to which the medical image belongs is judged, the ratio is used as an abnormal rate to carry out risk prompt.

The abnormal degrees corresponding to the ratios (i.e., different abnormal rates) of different sizes are different, so that prompting of different risk contents can be performed, for example, for the ellipsoidal band and/or the retinal outer limiting membrane, the ratio when the ellipsoidal band and/or the retinal outer limiting membrane are broken is smaller, which indicates that the severity of the abnormal eye is small, and at this time, the ellipsoidal band and/or the retinal outer limiting membrane can be prompted to be damaged or the disease is not severe.

When the target image slices with the fractures are continuous (namely, the target slices belong to the same continuous slice group) and the quantity is large, the fracture area of the target object is large, the severity of the abnormity is large, and the server carries out risk prompt according to the ratio between the maximum number of slices in the group and the total number of the slices.

As an example, the medical image is taken as an OCT image of an eye position for illustration, and it is assumed that the OCT image is sliced into 16 pieces from left to right, resulting in 16 OCT slices; grouping the OCT slices into a first image segmentation network, and when the first image segmentation network outputs 0, indicating that the ellipsoid zone and/or the retina outer limiting membrane are continuous in the OCT slices, namely the ellipsoid zone and/or the retina outer limiting membrane are normal; output 1 indicates that the ellipsoidal zone and/or the retinal outer limiting membrane are broken in the OCT slice, that is, the ellipsoidal zone and/or the retinal outer limiting membrane are abnormal, and the following estimation result p is obtained [0,0,0,0,0,0, 1,1,1,0,1,1,1,1,0 ]. Thus, a continuous slice group in which OCT slices are continuously abnormal can be obtained, and the number of abnormal slices in each group abn _ continuous is obtained by counting the number of continuous abnormalities [3,4 ]. Meaning that there are 2 consecutive abnormal slices in the current result, with the number of slices in each group being 3 and 4, respectively. The maximum anomaly rate can be obtained by dividing the maximum value in the abn _ continuos array, i.e. 4, by the number of slices of the OCT slice, i.e. 16, and the maximum anomaly rate calculated in the above example is 4/16 ═ 0.25, and the calculation formula is as follows:

the len (p) is the slice number of the OCT slice, and the risk of a large-scale abnormal area can be quantitatively prompted by calculating the maximum abnormal rate, so that the missed diagnosis is avoided.

In the above embodiment, when the target object is broken in the target image slice, the ratio between the number of slices of the continuous slice and the number of slices of the image slice sequence is calculated, the abnormal degree of the target object can be known by using the ratio as the abnormal rate of the target object, and the problem of erroneous judgment or missed judgment of the abnormal degree can be avoided by performing risk prompt according to the abnormal rate.

In one embodiment, as shown in fig. 8, the method may further include:

and S802, slicing the medical image sample to obtain a sample slice sequence.

The medical image sample may be an image obtained by non-invasively acquiring a human body or a part of a human body through medical equipment, or an image obtained by image-acquiring other objects through medical equipment, where the other objects may include various animals, plants, microorganisms, and the like.

In one embodiment, the server, when obtaining the medical image sample, normalizes the medical image sample based on a mean and a variance of the medical image sample; and carrying out image overturning on the processed medical image sample to obtain an overturned medical image sample. S802 may specifically include: and the server slices the turned medical image sample to obtain a sample slice sequence.

The manner of performing image processing, slice processing, and grouping processing on the medical image sample, and acquiring the medical image sample may refer to S202 of the above-described embodiment.

And S804, grouping the sample slices in the sample slice sequence into a first image classification network.

And S806, extracting first training image features of the target object in each sample slice of each group through a first feature extractor in the first image classification network.

S808, performing target object continuity calculation on the extracted first training image features through a first classifier in the first image classification network to obtain a first target continuity probability; and determining the target communication rate of the target object in the sample slice sequence based on the accumulated sum of the difference values between the first target continuous probabilities and the total slice amount of the sample slice sequence.

In the above embodiments, reference may be made to S204 to S208 in the above embodiments for S806 to S808.

And S810, calculating a first loss value between the target connectivity rate and the reference connectivity rate.

Wherein, the reference communication rate may be a communication rate calculated by a gold standard. In the medical field, the gold standard may refer to the most reliable method recognized by the clinical medical field, for example, a professional labels a target object in a medical image sample through a precision instrument, and then calculates the communication rate; further, the connectivity obtained by other reliable methods may be used.

In one embodiment, the server may calculate a first loss value between the target connectivity rate and the reference connectivity rate using an objective loss function, which may be any one of an L2 loss function, a Mean Squared Error (Mean Squared Error) function, and a cross entropy loss function.

Next, the above-described loss function will be explained:

1) the L2 loss function, also called the Least Squares Error (LSE), by which the sum of squares of the differences between the target communication rate f (xi) and the reference communication rate (Yi) can be calculated as the loss value between the target communication rate f (xi) and the reference communication rate (Yi). Wherein the L2 loss function is as follows:

2) the mean square error function firstly calculates the square of the difference between the real value and the estimated value, then sums the squares, and then calculates the average value, namely the target connectivity yi and the reference connectivity can be calculated by the mean square error function

The sum of squares of the differences between them, and then the average of the respective sums of squares is calculated as the loss value between the target communication rate f (xi) and the reference communication rate (Yi). Wherein the mean square error function is as follows:

3) the cross entropy loss function represents the difference between the true probability distribution p (xi) and the predicted probability distribution q (xi), and a smaller value represents a better predicted result. Wherein the cross entropy loss function is as follows:

s812, adjusting the parameters of the first feature extractor and the parameters of the first classifier according to the first loss value.

The parameters include convolutional layer parameters w and bias parameters b.

In one embodiment, the server may back-propagate the first loss value to network layers of the first feature extractor and the first classifier in the first image classification network, the network layers including the convolutional layer, the pooling layer, the normalization-activation layer (i.e., the network layer formed by combining the normalization layer with the ReLu function), the fully-connected layer, and the Softmax layer shown in fig. 4, obtaining gradients for parameters of the layers; and adjusting parameters of each network layer according to the gradient until the first image classification network converges.

The ReLu function is a linear rectification function, and can be used as an excitation function in the first image classification network, and an input vector from a previous neural network layer (i.e., a pooling layer) is normalized by the normalization-activation layer, and then a normalized vector x is operated by the ReLu function in the normalization-activation layer, where the ReLu function is f (x) max (0, x).

The Softmax layer may be a network layer constructed based on a Softmax function, and features input by each neuron in the fully-connected layer may be respectively mapped to values between 0 and 1 (the values are predicted communication rates), and the cumulative sum of the communication rates corresponding to each neuron is 1. After the connectivity corresponding to each neuron is obtained, a first classifier in the first image classification network may select a result corresponding to the maximum connectivity as a classification category to which the medical image belongs.

In an embodiment, before the first image classification network is trained, the second image classification network may be trained, and specifically, the training may be performed in two stages, that is, the first stage trains the second image classification network, and after the training of the second image classification network is completed, the second stage starts, that is, trains the first image classification network, as shown in fig. 9. S804 to S812 belong to a second stage, and the following training process belongs to a first stage, wherein the training step of the first stage specifically is:

the server sequentially inputs the sample slices in the sample slice sequence into a second image classification network; extracting a second training image feature of the target object in the input sample slice through a second feature extractor in a second image classification network; performing target object continuity calculation on the extracted second training image features through a second classifier in a second image classification network to obtain a second target continuity probability; calculating a second loss value between the second target continuity probability and the reference continuity rate; adjusting parameters of the second feature extractor and parameters of the second classifier according to the second loss value; and taking the adjusted second feature extractor as a first feature extractor in the first image classification network.

In one embodiment, the method further comprises: and respectively initializing parameters in the first image classification network and the second image classification network by the server based on the normal distribution of the target mean value and the target variance.

In the above embodiment, a sample slice sequence is obtained by slicing a medical image sample, and a first feature extractor in the first image classification network extracts a first training image feature of a target object in each image slice of the sample slice sequence, so that the amount of computation for extracting the first training image feature is reduced, and the rate of feature extraction is increased. A first classifier in the first image classification network sequentially determines first target continuous probability of a target object in each sample slice according to extracted first training image features, and determines target communication rate in a target object sample slice sequence based on the accumulated sum of difference values between the first target continuous probabilities of adjacent sample slices and the total slice amount of the sample slice sequence, so that the first image classification network learns the correlation between different sample slices, the first image classification network can extract correlation information between image slices of different medical images, the performance of the first image classification network is improved, and the accuracy of medical image classification is improved.

In one embodiment, as shown in fig. 10, an image classification network processing method is provided, which can be applied to at least one of the terminal 102, the server 104 and the medical device 108 in fig. 1, and is exemplified by the application of the method to the server 104 in fig. 1, and includes the following steps:

and S1002, slicing the medical image sample to obtain a sample slice sequence.

In one embodiment, the server, when obtaining the medical image sample, normalizes the medical image sample based on a mean and a variance of the medical image sample; and carrying out image overturning on the processed medical image sample to obtain an overturned medical image sample. S1002 may specifically include: and the server slices the turned medical image sample to obtain a sample slice sequence.

And S1004, sequentially inputting the sample slices in the sample slice sequence into a second image classification network.

And S1006, extracting a second training image feature of the target object in the input sample slice through a second feature extractor in the second image classification network.

The manner of extracting the second training image features in S1006 may refer to the manner of extracting the image features in S204 in the above embodiment.

And S1008, performing target object continuity calculation on the extracted second training image features through a second classifier in a second image classification network to obtain a second target continuity probability.

The manner of performing the target object continuity calculation in S1008 may refer to the manner of determining the continuity probability in S206 in the above embodiment.

And S1010, calculating a second loss value between the second target continuous probability and the reference continuous rate.

In one embodiment, the server may calculate the first Loss value between the second target continuity probability and the reference continuity rate using an objective Loss function, which may be any one of an L2 Loss function, Mean Squared Error (Mean Squared Error), cross entropy Loss function, and Focal Loss function.

And S1012, adjusting the parameters of the second feature extractor and the parameters of the second classifier according to the second loss value.

In one embodiment, the server may back-propagate the second penalty value to network layers of a second feature extractor and a second classifier in a second image classification network, the network layers including the convolutional layer, the pooling layer, the normalization layer + ReLu, the fully-connected layer, and the Softmax layer shown in fig. 4, obtaining gradients for parameters of the layers; and adjusting the parameters of each network layer according to the gradient until the second image classification network converges.

And S1014, taking the adjusted second feature extractor as a first feature extractor in the first image classification network to obtain the first image classification network comprising the first feature extractor and the first classifier.

Wherein, S1004 to S1014 belong to a first stage of training, and after finishing the training of the first stage, the trained second feature extractor is used as a first feature extractor in the first image classification network, that is, the first image classification network adopts or shares the second feature extractor of the second image classification network; the first feature extractor is combined with the first classifier to obtain a first image classification network.

S1016, the sample slice groups in the sample slice sequence are input into the first image classification network.

S1018, extracting, by a first feature extractor in the first image classification network, a first training image feature of the target object in each sample slice of each group.

S1020, performing target object continuity calculation on the extracted first training image features through a first classifier in the first image classification network to obtain a first target continuity probability; and determining the target communication rate of the target object in the sample slice sequence based on the accumulated sum of the difference values between the first target continuous probabilities and the total slice amount of the sample slice sequence.

In the above, reference may be made to S204 to S208 in the above embodiments for S1016 to S1020.

S1022, a first loss value between the target connectivity rate and the reference connectivity rate is calculated.

And S1024, adjusting parameters of the first feature extractor and parameters of the first classifier according to the first loss value.

Wherein S1016-S1024 belong to the second stage of training. The steps S1022 to S1024 refer to S810 to S812 in the above embodiment.

It should be understood that although the steps in the flowcharts of fig. 2, 5, 7, 8, 10 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2, 5, 7, 8, and 10 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternatingly with other steps or at least some of the other steps or stages.

As an example, the medical image is taken as an OCT image for explanation: firstly, a training set is obtained, wherein the training set can be composed of different OCT images, the OCT images are used for training an image classification network, the training mode is as shown in fig. 9, and model training is mainly performed in two stages:

1) first stage

The feature extractors and classifiers in the image classification network are trained with slices of the OCT image (called OCT slices) to determine if there is a fracture in the EZ and EML.

The feature extractor is used as a backbone of an image classification network, and ResNet101 can be selected; the classifier may be implemented using a fully connected layer.

2) Second stage

The method comprises the steps of adopting a backbone network in a first stage as a feature extractor of an image classification network in a second stage, combining the feature extractor with a classifier realized based on LSTM to obtain a new image classification network, and then training the new image classification network by utilizing OCT slices to increase the information correlation degree among the OCT slices.

It should be noted that the image classification network of the second stage is directly used in the testing stage for testing.

Next, a technique employed in the present embodiment will be explained:

(1) connectivity and loss of connectivity

For the image classification networks of the two training phases, cross-entropy loss (softmax cross-entropy loss) is adopted in the image classification network of the first phase for training the image classification network of the first phase. In the image classification network in the first stage, a loss function with connectivity as an evaluation index is adopted, so that the information correlation degree between OCT slices can be increased.

Where EZ and ELM connectivity are defined as follows, assuming that there are N input consecutive image slices in the second stage, the connectivity can be calculated by the following connectivity calculation formula:

wherein Diff¹Representing the difference between successive probabilities of adjacent OCT slices, Diff¹The upper right 1 indicates the difference between adjacent OCT slices, in case of Diff²Then, it is shown in Diff¹On the basis of which the Diff is calculated once more¹I.e. a second order differential. Diff may be used when the number of consecutive slices input in the second stage is less than 10¹The requirements are met; when the number of consecutive slices input in the second stage is greater than 10, Diff may be used². Then, the second stage of training is performed using the loss of L2 as an auxiliary supervision loss, using the connectivity as an evaluation index, and the calculation function of the loss of connectivity (i.e., the loss value of connectivity) is as follows:

wherein p is_cPredicting the resulting connectivity, y, through the image classification network of the second stage_cThe communication rate calculated according to the gold standard.

(2) Maximum rate of abnormality

An estimate of the degree of anomaly, referred to as the maximum anomaly rate, is also given on top of the results of the EZ and ELM anomaly detections. Assuming that an OCT image is segmented into 16 OCT slices from left to right, obtaining 16 OCT slices, and if the network outputs 0, indicating that EZ and ELM are normal; if the network output 1 indicates that EZ and ELM are abnormal, the following estimation result p is obtained [0,0,0,0,0,0, 1,1,1,1,0 ]. By counting the number of consecutive abnormalities, abn _ continuous ═ 3,4 can be obtained, which indicates that there are 3 consecutive OCT slice abnormalities, and 4 consecutive OCT slice abnormalities. The method for counting the number of consecutive anomalies is as follows:

x＝np.array([1,0,0,0,0,1,0,1,0,0,0,0,0])

point_1＝np.squeeze(np.array(np.where(x＝＝1)))

if x[-1]＝＝0:

last_point＝np.array([len(x)])

point_1＝np.concatenate((point_1,last_point),0)

for i in range(len(point_1)-1):

print(point_1[i+1]-point_1[i]-1)

the maximum value in the abn _ continuous array, i.e. 4, is taken out and divided by the total length of the vector p, which is the number 16 of OCT slices, so as to obtain the required maximum abnormal rate, and the calculation formula is as follows:

by means of the maximum abnormal rate, the risk that large-scale abnormal areas of EZ and ELM occur can be quantitatively prompted, and missed diagnosis is avoided.

(3) Network training

2) Parameter initialization

In different image classification networks of two stages, network parameters of the image classification networks adopt parameters pre-trained on an ImageNet data set by a ResNet101 network; in addition, all network parameters are initialized with a gaussian distribution with variance of 0.01 and mean of 0.

2) Data preprocessing and training

The OCT images are normalized, wherein the normalization process may be performed as follows: the image mean is subtracted from each element in the matrix of the OCT image, and then the absolute value of the difference is divided by the image variance. After the normalization process is performed, a random horizontal flipping operation is performed on the OCT image after the normalization process. In this embodiment, a gradient descent method based on Adam is adopted to solve the convolutional layer parameters w and the bias parameters b in the image classification network, and a training strategy of war up and cosine learning rate attenuation is adopted. The training process of the network is supervised by a loss function. And when the loss value of the image classification network is greater than or equal to the threshold value, adjusting the parameters of the image classification network, and continuing training until the loss value is less than the loss threshold value. And when the loss value of the image classification network is smaller than the loss threshold value or the loss value is converged, stopping training to obtain the trained image classification network.

By implementing the scheme of the embodiment, the following technical effects can be achieved:

1) a brand-new EM and ELM abnormity detection scheme based on deep learning is provided, and whether EM and ELM are abnormal or not can be effectively detected.

2) A two-stage training mode is provided, the network is subjected to coarse training in the first stage, and fine training is performed in the second stage to enhance the relevance among the slices, so that the detection accuracy is improved.

3) The communication rate and the maximum abnormal rate are initially created, quantitative nAMD (neovascular age-related macular degeneration) risk analysis is provided, the communication is used as an evaluation index, and the accuracy of network training can be further improved by using L2 loss as a loss function.

In one embodiment, as shown in fig. 11, there is provided a medical image classification apparatus including: a processing module 1102, an extraction module 1104, a determination module 1106, and a determination module 1108; wherein:

a processing module 1102, configured to perform slice processing on a medical image to obtain an image slice sequence;

an extraction module 1104, configured to extract, from each image slice in the sequence of image slices, an image feature of the object in the image slice;

a determining module 1106, configured to sequentially determine continuous probabilities of the target object in each image slice according to the image features extracted from each image slice; determining the communication rate of the target object in the image slice sequence based on the accumulated sum of the difference values between the continuous probabilities corresponding to each adjacent image slice in the image slice sequence and the total slice amount of the image slices in the image slice sequence;

and the judging module 1108 is configured to judge the classification category to which the medical image belongs according to the connectivity.

In one embodiment, the processing module 1102 is further configured to determine a slice width and a step size of the step-by-step movement during the slice processing; gradually cutting out image slices with slice widths in the medical image according to step lengths to obtain an image slice sequence; or, determining a region of interest in the medical image, and gradually cutting out image slices of slice widths in the region of interest according to step sizes to obtain an image slice sequence.

In one embodiment, the extracting module 1104 is further configured to group the image slices in the image slice sequence according to a slice order, so as to obtain at least two groups of image slices; and respectively carrying out feature extraction on the image slices of each group through a first feature extractor in the first image classification network to obtain the image features of the target object in the image slices of each group.

In one embodiment, the determining module 1106 is further configured to perform anomaly identification on the image features respectively extracted from the image slices through a first classifier in a first image classification network, so as to obtain an identification result indicating whether the target object is broken in each image slice; determining the continuous probability of the target object in each image slice based on the identification result; or determining the fracture probability of the target object in each image slice according to the identification result, and obtaining the continuous probability according to the fracture probability.

In one embodiment, as shown in fig. 12, the apparatus further comprises: a marking module 1110 and a display module 1112; wherein:

a determining module 1106, configured to determine a fracture position of the object in the target image slice when the object is determined to be fractured in the target image slice of the image slice sequence according to the recognition result;

a marking module 1110 for marking a location area in the medical image matching the fracture location;

a display module 1112, configured to display the labeled medical image and the classification category after determining the classification category to which the medical image belongs.

In one embodiment, as shown in fig. 12, the apparatus further comprises: a prompt module 1114, wherein:

a prompt module 1114, configured to detect continuity of sequence numbers between multiple target image slices when it is determined that a target object has a break in the multiple target image slices of the image slice sequence according to the recognition result; determining the number of the continuous slices in the plurality of target image slices according to the continuity; determining the ratio of the number of slices to the total number of slices; and after the classification category to which the medical image belongs is judged, the ratio is used as the abnormal rate to carry out risk prompt.

In one embodiment, the prompt module 1114 is further configured to determine a set of consecutive slices in the plurality of target image slices according to the continuity; acquiring the number of slices in each continuous slice group; the number of slices comprises the number of slices in the group; selecting the largest number of the group internal slices from the obtained number of the group internal slices; and taking the ratio between the maximum number of the intra-group slices and the total number of the slices as an abnormal rate to carry out risk prompt.

In one of the embodiments, the slice order includes an order in which the medical images are sliced;

the extracting module 1104 is further configured to sort the image slices in the image slice sequence according to a slice order; determining the number of slices in the group; and grouping the image slices in the ordered image slice sequence according to the number of slices in the group.

In one embodiment, as shown in fig. 12, the apparatus further comprises: an input module 1116, a calculation module 1118, and an adjustment module 1120, wherein:

the processing module 1102 is further configured to perform slicing processing on the medical image sample to obtain a sample slice sequence;

an input module 1116 for inputting a group of sample slices in the sequence of sample slices into the first image classification network;

the extracting module 1104 is further configured to extract, by a first feature extractor in the first image classification network, first training image features of the target object in each sample slice of each group;

the determining module 1106 is further configured to perform target object continuity calculation on the extracted first training image features through a first classifier in the first image classification network to obtain a first target continuity probability; determining a target communication rate of the target object in the sample slice sequence based on the accumulated sum of the difference values between the first target continuous probabilities and the total slice amount of the sample slice sequence;

a calculation module 1118 configured to calculate a first loss value between the target connectivity rate and the reference connectivity rate;

an adjusting module 1120, configured to adjust the parameter of the first feature extractor and the parameter of the first classifier according to the first loss value.

In one embodiment, as shown in fig. 12, the apparatus further comprises: a flipping module 1122; wherein:

the processing module 1102 is further configured to, when obtaining the medical image sample, perform normalization processing on the medical image sample based on a mean and a variance of the medical image sample;

the flipping module 1122 is configured to flip the processed medical image sample to obtain a flipped medical image sample;

the processing module 1102 is further configured to slice the flipped medical image sample to obtain a sample slice sequence.

In one embodiment, the input module 1116 is further configured to sequentially input the sample slices in the sample slice sequence into the second image classification network;

the extraction module 1104 is further configured to extract, by a second feature extractor in a second image classification network, a second training image feature of the target object in the input sample slice;

the determining module 1106 is further configured to perform target object continuity calculation on the extracted second training image features through a second classifier in a second image classification network to obtain a second target continuity probability;

a calculating module 1118, further configured to calculate a second loss value between the second target continuity probability and the reference continuity rate;

the adjusting module 1120 is further configured to adjust the parameters of the second feature extractor and the parameters of the second classifier according to the second loss value; and taking the adjusted second feature extractor as a first feature extractor in the first image classification network.

In one embodiment, the rate of communication is a first order rate of communication; a determining module 1106, configured to determine a difference between adjacent connectivity rates in a first-order connectivity rate; summing the absolute values of the difference values to obtain an accumulated value; determining the second-order communication rate of the target object in the image slice sequence according to the accumulated value and the number of the first-order communication rates;

the determining module 1108 is further configured to determine a classification category to which the medical image belongs according to the second-order connectivity.

In one embodiment, the apparatus further comprises:

and the initialization module is used for respectively initializing parameters in the first image classification network and the second image classification network based on the normal distribution of the target mean value and the target variance.

For the specific definition of the medical image classification apparatus, reference may be made to the above definition of the medical image classification method, which is not described herein again. The modules in the medical image classification apparatus can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, as shown in fig. 13, there is provided an image classification network processing apparatus including: processing module 1302, input module 1304, extraction module 1306, determination module 1308, calculation module 1310, and adjustment module 1312; wherein:

a processing module 1302, configured to perform slicing processing on a medical image sample to obtain a sample slice sequence;

an input module 1304 for inputting groups of sample slices in the sequence of sample slices into a first image classification network;

an extracting module 1306, configured to extract, by a first feature extractor in the first image classification network, a first training image feature of the target object in each sample slice of each group;

a determining module 1308, configured to perform target object continuity calculation on the extracted first training image feature through a first classifier in the first image classification network, so as to obtain a first target continuity probability; and determining a target communication rate of the target object in the sample slice sequence based on the accumulated sum of the difference values between the first target continuous probabilities and the total slice amount of the sample slice sequence;

a calculating module 1310 for calculating a first loss value between the target connectivity rate and a reference connectivity rate;

an adjusting module 1312 is configured to adjust the parameters of the first feature extractor and the parameters of the first classifier according to the first loss value.

In one embodiment, as shown in fig. 14, the apparatus further comprises: a flip module 1314; wherein:

a processing module 1302, configured to, when obtaining the medical image sample, perform normalization processing on the medical image sample based on a mean and a variance of the medical image sample;

the overturning module 1314 is configured to perform image overturning on the processed medical image sample to obtain an overturned medical image sample;

the processing module 1302 is further configured to slice the flipped medical image sample to obtain a sample slice sequence.

In one embodiment, the input module 1304 is further configured to sequentially input the sample slices in the sample slice sequence into the second image classification network;

the extracting module 1306 is further configured to extract, by a second feature extractor in the second image classification network, a second training image feature of the target object in the input sample slice;

the third determining module 1308 is further configured to perform target object continuity calculation on the extracted second training image features through a second classifier in the second image classification network, so as to obtain a second target continuity probability;

a calculating module 1310, further configured to calculate a second loss value between the second target continuity probability and the reference continuity rate;

an adjusting module 1312, further configured to adjust the parameters of the second feature extractor and the parameters of the second classifier according to the second loss value; and taking the adjusted second feature extractor as a first feature extractor in the first image classification network.

In one embodiment, the apparatus further comprises:

For specific definition of the image classification network processing device, refer to the above definition of the image classification network processing method, which is not described herein again. The modules in the image classification network processing device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, a terminal or a medical device, and the computer device is a server for example, and the internal structure diagram thereof may be as shown in fig. 15. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device may be used for storing medical images. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a medical image classification method or an image classification network processing method.

Those skilled in the art will appreciate that the architecture shown in fig. 15 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.

In an embodiment, a computer-readable storage medium is provided, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

In one embodiment, a computer program product or computer program is provided that includes computer instructions stored in a computer-readable storage medium. The computer instructions are read by a processor of a computer device from a computer-readable storage medium, and the computer instructions are executed by the processor to cause the computer device to perform the steps in the above-mentioned method embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method of medical image classification, the method comprising:

slicing the medical image to obtain an image slice sequence;

2. The method of claim 1, wherein the slicing the medical image to obtain the sequence of image slices comprises:

determining the width of a slice and the step length of gradual movement during slice processing;

gradually cutting out the image slices with the slice width in the medical image according to the step length to obtain an image slice sequence; alternatively, the first and second electrodes may be,

and determining a region of interest in the medical image, and gradually cutting out the image slices with the slice width in the region of interest according to the step length to obtain an image slice sequence.

3. The method of claim 1, wherein the extracting, from each image slice within the sequence of image slices, an image feature of an object in the image slice comprises:

grouping the image slices in the image slice sequence according to a slice sequence to obtain at least two groups of image slices;

and respectively carrying out feature extraction on the image slices of each group through a first feature extractor in a first image classification network to obtain the image features of the target object in the image slices of each group.

4. The method of claim 3, wherein sequentially determining the sequential probability of the object in each of the image slices according to the image features extracted from each of the image slices comprises:

performing anomaly identification on the image features extracted from each image slice through a first classifier in the first image classification network to obtain an identification result indicating whether the target object is broken in each image slice;

determining a continuous probability of the object in each of the image slices based on the recognition result; alternatively, the first and second electrodes may be,

and determining the fracture probability of the target object in each image slice according to the identification result, and obtaining the continuous probability according to the fracture probability.

5. The method of claim 4, further comprising:

when the target object is determined to be broken in the target image slice of the image slice sequence according to the identification result, determining the breaking position of the target object in the target image slice;

marking a location area in the medical image that matches the fracture location;

and after the classification category to which the medical image belongs is judged, displaying the marked medical image and the classification category.

6. The method of claim 4, further comprising:

when the target object is determined to have fracture in a plurality of target image slices of the image slice sequence according to the identification result, detecting the continuity of the sequence numbers among the plurality of target image slices;

determining a slice number of consecutive slices of the plurality of target image slices according to the continuity;

determining a ratio of the number of slices to the total number of slices;

and after the classification category to which the medical image belongs is judged, the ratio is used as an abnormal rate to carry out risk prompt.

7. The method of claim 6, wherein said determining a slice number of successive slices of the plurality of target image slices from the continuity comprises:

determining a set of consecutive slices in the plurality of target image slices according to the continuity;

acquiring the number of slices in each continuous slice group; the number of slices comprises the number of slices in the group;

selecting the largest number of the group internal slices from the obtained number of the group internal slices;

the risk prompting by taking the ratio as the abnormal rate comprises the following steps:

and taking the ratio between the number of the slices in the group based on the maximum number and the total number of the slices as an abnormal rate to carry out risk prompt.

8. The method of claim 3, wherein the slice order comprises an order in which the medical image is sliced;

the grouping of image slices in the sequence of image slices in slice order comprises:

ordering the image slices in the sequence of image slices in the slice order;

determining the number of slices in the group;

and grouping the image slices in the sequenced image slice sequence according to the number of the slices in the group.

9. The method according to claim 3 or 4, characterized in that the method further comprises:

slicing a medical image sample to obtain a sample slice sequence;

inputting a grouping of sample slices in the sequence of sample slices into the first image classification network;

extracting first training image features of a target object in each sample slice of each group through a first feature extractor in the first image classification network;

10. The method according to any one of claims 1 to 8, wherein the rate of communication is a first order rate of communication; the method further comprises the following steps:

determining a difference between adjacent connectivity rates in the first order connectivity rate;

summing the absolute values of the difference values to obtain an accumulated value;

determining a second-order communication rate of the target object in the image slice sequence according to the accumulated value and the number of the first-order communication rates;

the determining the classification category to which the medical image belongs according to the communication rate comprises:

and judging the classification category of the medical image according to the second-order communication rate.

11. An image classification network processing method, characterized in that the method comprises:

slicing a medical image sample to obtain a sample slice sequence;

12. A medical image classification apparatus, characterized in that the apparatus comprises:

13. An image classification network processing apparatus, characterized in that the apparatus comprises:

14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 11 when executing the computer program.

15. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 11.