CN108961296B

CN108961296B - Fundus image segmentation method, fundus image segmentation device, fundus image segmentation storage medium and computer equipment

Info

Publication number: CN108961296B
Application number: CN201810825633.4A
Authority: CN
Inventors: 蒋忻洋; 王子龙; 孙星; 王睿
Original assignee: Tencent Healthcare Shenzhen Co Ltd
Current assignee: Tencent Healthcare Shenzhen Co Ltd
Priority date: 2018-07-25
Filing date: 2018-07-25
Publication date: 2020-04-14
Anticipated expiration: 2038-07-25
Also published as: CN111192285B; CN111192285A; CN108961296A

Abstract

The application relates to a fundus image segmentation method, a fundus image segmentation device, a storage medium and a computer device, wherein the method comprises the following steps: acquiring a fundus image to be segmented; dividing a plurality of pixel blocks from the fundus image; determining a plurality of probability image blocks respectively corresponding to different focus categories according to each pixel block; each color value in the probability picture block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category; determining the focus category of each pixel point in the fundus image according to the probability picture block; and segmenting a focus area from the fundus image according to the focus category to which each pixel point of the fundus image belongs. The scheme provided by the application can improve the accuracy of fundus image segmentation.

Description

Fundus image segmentation method, fundus image segmentation device, fundus image segmentation storage medium and computer equipment

Technical Field

The present application relates to the field of image segmentation technologies, and in particular, to a method, an apparatus, a storage medium, and a computer device for segmenting an eye fundus image.

Background

With the development of image processing technology, image segmentation technology is beginning to be applied to the field of fundus image segmentation, and whether focus characteristics appear on human eyes can be automatically detected by performing segmentation processing on fundus images.

At present, researchers at home and abroad propose various fundus image segmentation algorithms, the commonly used fundus image segmentation algorithm is a segmentation algorithm based on blood vessel tracking, and the segmentation algorithm is realized by the following steps: a local operator acts on a certain initial point known as a blood vessel, parameters such as the central line, the direction and the radius of the blood vessel are automatically tracked by an algorithm, and image segmentation is carried out on the fundus image according to the parameters. However, in the segmentation algorithm based on blood vessel tracking, a tracking error is liable to occur when a blood vessel branch ignition intersection is encountered, thereby affecting the accuracy of fundus image segmentation.

Disclosure of Invention

In view of the above, it is necessary to provide a fundus image segmentation method, apparatus, storage medium, and computer device for solving the technical problem of low accuracy of fundus image segmentation due to the segmentation algorithm using blood vessel tracking.

A fundus image segmentation method, comprising:

acquiring a fundus image to be segmented;

dividing a plurality of pixel blocks from the fundus image;

determining a plurality of probability image blocks respectively corresponding to different focus categories according to each pixel block; each color value in the probability picture block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category;

determining the focus category of each pixel point in the fundus image according to the probability picture block;

and segmenting a focus area from the fundus image according to the focus category to which each pixel point of the fundus image belongs.

A fundus image segmentation apparatus, comprising:

the fundus image acquisition module is used for acquiring a fundus image to be segmented;

a pixel block division module for dividing a plurality of pixel blocks from the fundus image;

a probability image block determining module for determining a plurality of probability image blocks respectively corresponding to different focus categories according to each pixel block; each color value in the probability picture block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category;

the focus category determining module is used for determining the focus category of each pixel point in the fundus image according to the probability picture block;

and the segmentation module is used for segmenting a focus area from the fundus image according to the focus category to which each pixel point of the fundus image belongs.

A storage medium stores a computer program that, when executed by a processor, causes the processor to execute the steps of the fundus image segmentation method.

A computer apparatus comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of the fundus image segmentation method.

The fundus image segmentation method, the fundus image segmentation device, the storage medium and the computer equipment divide a plurality of pixel blocks from the acquired fundus image to be segmented, and process the divided pixel blocks, so that the whole fundus image is prevented from being processed, and the calculation amount is reduced. And processing the divided pixel blocks to obtain a plurality of probability image blocks respectively corresponding to different focus categories, and determining the focus category to which each pixel point in the fundus image belongs according to the probability image blocks, thereby realizing focus feature identification of each pixel in the fundus image. According to the focus category to which each pixel point of the fundus image belongs, a focus region is segmented from the fundus image, so that focus segmentation of the fundus image is realized, and the accuracy of fundus image segmentation is improved.

Drawings

FIG. 1 is a diagram showing an environment in which a fundus image segmentation method is applied in one embodiment;

FIG. 2 is a flowchart illustrating a fundus image segmentation method according to an embodiment;

FIG. 3 is a schematic view of an interface of a fundus image and a corresponding lesion area image in one embodiment;

FIG. 4 is a flowchart of a step of dividing a fundus image in one embodiment;

FIG. 5 is a flowchart illustrating the steps of training a machine learning model in one embodiment;

FIG. 6 is a flowchart illustrating the steps of processing a training pixel block in one embodiment;

FIG. 7 is a flowchart illustrating the steps of performing image enhancement processing on a reference pixel block and adjusting parameters of a machine learning model in one embodiment;

FIG. 8 is a flowchart illustrating steps of constructing a machine learning model in one embodiment;

FIG. 9 is a flowchart illustrating the steps of training a machine learning model in one embodiment;

FIG. 10 is a flowchart showing a step of segmenting a fundus image in one embodiment;

fig. 11 is a block diagram showing the configuration of a fundus image segmentation apparatus in one embodiment;

fig. 12 is a block diagram showing the configuration of a fundus image segmentation apparatus in another embodiment;

FIG. 13 is a block diagram of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

Fig. 1 is an application environment diagram of a fundus image segmentation method in one embodiment. Referring to fig. 1, the fundus image segmentation method is applied to a fundus image segmentation system. The fundus image segmentation system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected through a network. The terminal 110 may specifically be a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The server 120 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers.

In one embodiment, as shown in fig. 2, a fundus image segmentation method is provided. The embodiment is mainly illustrated by applying the method to the terminal 110 in fig. 1. Referring to fig. 2, the fundus image segmentation method specifically includes the steps of:

s202, fundus images to be segmented are acquired.

Here, the fundus refers to the tissue of the posterior part in the eyeball, i.e., the inner membrane of the eyeball. Many lesions are generally reflected on the fundus, and changes in the state of the fundus to some extent reflect the degree of change in some organs. Therefore, whether or not the corresponding lesion appears can be determined by the analysis of the fundus image. Retinal arteriosclerosis can be seen from the fundus of the eye of a hypertensive patient, and capillary hemangioma, bleeding spots, exudates and the like can be seen from the fundus of the eye of a diabetic patient.

In one embodiment, the terminal establishes a connection with a photographing apparatus by which a fundus image to be segmented is acquired. Or the terminal receives the selection instruction, and selects the corresponding fundus image stored in the terminal memory according to the selection instruction.

S204, a plurality of pixel blocks are divided from the fundus image.

Where the pixel blocks are images to be composed of a certain number of pixels in the fundus image, such as 256 × 256-resolution pixel blocks.

In one embodiment, the terminal may crop the pixel block at a preset size on the fundus image. Every time one pixel block is cut, the starting point of the cut is moved forward by a distance ensuring that each region of the fundus image is acquired, thereby obtaining a plurality of pixel blocks. Wherein the distance of movement may be less than or equal to the size of the pixel block. For example, if the size of the clipped pixel block is m × n, in order to ensure that each region of the fundus image is acquired, the distance of movement may be less than or equal to m at the time of horizontal movement when dividing the fundus image. In the case of vertical movement, the distance of movement may be less than or equal to n.

And S206, determining a plurality of probability image blocks respectively corresponding to different focus categories according to each pixel block, wherein each color value in the probability image blocks represents the probability that the corresponding pixel point in the pixel block belongs to each focus category.

The lesion category refers to a type of lesion that can be determined from the fundus image, such as hard effusion, microaneurysm, hemorrhage, and soft effusion. A probability tile may refer to a bitmap image, consisting of a single point of pixels (i.e., picture elements), which may be arranged and colored differently.

In one embodiment, the terminal inputs the pixel blocks obtained by division into a trained machine learning model, processes the input pixel blocks through the machine learning model, and calculates the probability that each pixel in each pixel block belongs to each focus category. And the terminal determines the color value of the probability picture block according to the calculated probability, and draws the corresponding probability picture block according to the color value to obtain a plurality of probability picture blocks respectively corresponding to different lesion classes.

The probability blocks corresponding to the probabilities belonging to different lesion categories have different colors, that is, different color values, for example, the color of the probability block belonging to the soft exudation may be red, and the color of the probability block belonging to the hard exudation may be yellow, etc.

For example, assume that a fundus image is divided into m small pixel blocks, and there are k kinds of lesion classes. The terminal inputs the m pixel blocks into the trained machine learning model for processing, the probability that each pixel point in each pixel block belongs to k kinds of focus categories is obtained through calculation, and k multiplied by m probability image blocks can be obtained.

The machine learning model may be a neural network classification model that is processed by: the classification layer in the neural network classification model is deleted, the input size is correspondingly adjusted, and the convolution layer is accessed in the last layer. The neural network classification model may include: a deep convolutional neural network model, or a deep full-lap machine network model, or other deep neural network models. The deep convolutional neural network model may be a ResNet101 network model. The deep full-volume machine network model may be a U-Net network model. Other deep neural network models may be, for example, Incepton-Resnet-V2, ResNext, NasNet, MobileNet, etc.

In one embodiment, S204 may specifically include: determining the corresponding characteristics of each pixel point in each pixel block; comparing the determined characteristics with characteristics of different focus categories to obtain the probability that each pixel point in each pixel block belongs to each focus category; determining color values of pixel points for synthesizing probability image blocks according to the probability; and synthesizing pixel points with color values into a plurality of probability image blocks according to different focus categories.

The feature corresponding to each pixel point may be a pixel feature, such as a color value.

In one embodiment, after obtaining the probability that each pixel point in each pixel block belongs to each focus category, the terminal determines whether the obtained probability is greater than or equal to a first probability threshold, and if so, indicates that the corresponding pixel point belongs to the corresponding focus type. Determining color values of the pixels for synthesizing the probability tiles according to the probabilities greater than or equal to the first probability threshold. If the color value is smaller than the first probability threshold, the corresponding color value is a colorless or black color value.

For example, assume there are k lesion classes, m1, m2 … mi … mk, respectively. The terminal respectively inputs one of the pixel blocks into a trained machine learning model for processing, and the probability that each pixel point in the input pixel block belongs to k focus categories is calculated to be p_m1 ⁿ、p_m2 ⁿ…p_mi ⁿ…p_mk ⁿWherein n is the number of pixels in the pixel block, p_mi ⁿIndicating the probability that the ith pixel belongs to the category of the mi lesion. If p is_mi ⁿWhen the probability of (d) is greater than or equal to the first probability threshold, it indicates that the ith pixel belongs to the mi-th lesion class according to p_mi ⁿColor values for pixels that synthesize a probability tile are determined.

And S208, determining the focus category of each pixel point in the fundus image according to the probability image block.

Wherein, the focus category includes but is not limited to the following: no focus, hard exudation, microaneurysms, hemorrhage and soft exudation. It is noted that no lesion is a special type of lesion category.

In one embodiment, S208 may specifically include: the terminal splices a plurality of probability image blocks corresponding to the same focus category to obtain fundus bitmaps corresponding to different focus categories respectively; determining the probability that pixel points at corresponding positions in the fundus bitmap belong to each focus category; and (4) attributing the pixel points of the corresponding positions to the focus categories of the corresponding maximum probability.

In one embodiment, after determining the probability that the pixel points at the corresponding positions in the fundus bitmap belong to each focus category, the terminal determines whether the probability is greater than or equal to a second probability threshold, and classifies the pixel points at the corresponding positions into the focus categories corresponding to the probability greater than or equal to the second probability threshold. When the fundus presents two focuses at the same time, the two focuses can be judged through the embodiment of the invention.

In an embodiment, the step of obtaining fundus bitmaps corresponding to different types of lesions by splicing a plurality of probability tiles corresponding to the same type of lesions by the terminal may specifically include: splicing a plurality of probability image blocks corresponding to each focus category according to the positions of the corresponding pixel blocks divided from the fundus image; determining an overlapping area between spliced probability images during splicing; and determining the average value of the color values of a plurality of pixel points at the same position in the overlapping region as the color value of the corresponding pixel point in the overlapping region, and obtaining the fundus bitmap corresponding to the corresponding focus category.

Because the size of the probability image block is larger than the step length of gradual movement when the pixel blocks are divided, the terminal can generate an overlapping area in the process of splicing a plurality of probability image blocks of the same focus category. The terminal determines the overlapping area between adjacent probability image blocks in the splicing process, and the average value of a plurality of pixel points at the same position in the overlapping area is obtained, so that the color of the overlapping area of the probability image blocks after splicing cannot generate mutation.

In one embodiment, the terminal receives a step instruction carrying a step to be moved step by step when dividing the pixel block, and determines the step to be moved step by step when dividing the pixel block from the fundus image according to the received step instruction. Wherein the step size is less than or equal to the size of the probability tile.

S210, dividing a focus area from the fundus image according to the focus category to which each pixel point of the fundus image belongs.

The focal region may refer to a region having a focal in the fundus image. The size of the focal region may be the same as the size of the pixel block described in the embodiment of the present invention, or may be the same as the size of the fundus image. The color value of each pixel point in the focus area can use different values according to different focus categories, so that different focus categories can be distinguished according to different colors. And the color value of each pixel point in the focal region is the prediction result of the machine learning model on the fundus image.

In one embodiment, after the terminal determines the probability that the pixel points at the corresponding positions in the fundus bitmap belong to each focus category, the pixel points with the highest probability belonging to the same positions are extracted, and the extracted pixel points are determined as focus pixel points.

For example, assume that there are k types of lesions, m1 and m2 … mk respectively, the number of fundus bitmaps obtained after stitching is k, and the number of corresponding pixels is p × q. For pixel points at each position in the fundus bitmap of k focus categories of m1 and m2 … mk, the terminal extracts the pixel point with the highest probability in the same position. As in the fundus bitmap position (x)_i,y_i) The probabilities corresponding to the k focus categories of m1 and m2 … mk are p respectively_m1 ⁱ、p_m2 ⁱ…p_mk ⁱIf p is_m1 ⁱIs the largest, then position (x)_i,y_i) The focus type corresponding to the pixel point of (a) is m1, and the position (x) is obtained from the fundus bitmap corresponding to m1_i,y_i) The pixel points are extracted as focus pixel points.

In one embodiment, S210 may specifically include: and the terminal respectively combines the extracted pixel points according to the extracted positions, and takes the combined image as a focus area after the fundus image is segmented.

In the above-described embodiment, the plurality of pixel blocks are divided from the fundus image to be divided, and the divided pixel blocks are processed, so that the entire fundus image is prevented from being processed, thereby reducing the amount of calculation. And processing the divided pixel blocks to obtain a plurality of probability image blocks respectively corresponding to different focus categories, and determining the focus category to which each pixel point in the fundus image belongs according to the probability image blocks, thereby realizing focus feature identification of each pixel in the fundus image. According to the focus category to which each pixel point of the fundus image belongs, a focus region is segmented from the fundus image, so that focus segmentation of the fundus image is realized, and the accuracy of fundus image segmentation is improved.

As an example, as shown in fig. 3, fig. 3(a) is a fundus image to be segmented, and fig. 3(b) is a lesion region image obtained after segmenting the fundus image, that is, a final lesion fundus bitmap. Dividing the fundus image to obtain a plurality of pixel blocks, respectively inputting the pixel blocks into a machine learning model, and determining a plurality of probability image blocks respectively corresponding to different focus categories according to each pixel block, wherein each color value in each probability image block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category; determining the focus category of each pixel point in the fundus image according to the probability picture block; the focal region is segmented from the fundus image according to the focal type to which each pixel point of the fundus image belongs, so that the focal region image of fig. 3(b) can be obtained.

In an embodiment, as shown in fig. 4, S204 may specifically include:

s402, determining the size of the pixel block to be divided.

In one embodiment, the terminal determines the size of the pixel block to be divided according to an input size operation instruction, wherein the size operation instruction carries the size of the pixel block. Or, the terminal obtains the size of the pixel block to be divided from a preset size. Alternatively, the terminal determines the size of the pixel block to be divided in a preset ratio according to the size of the fundus image to be divided, for example, the size of the fundus image is 100 × 100, and the preset ratio is 0.1, then the size of the pixel block to be divided is 10 × 10.

S404, determining a step size to be moved step by step when pixel blocks are divided step by step from the fundus image; the step size is smaller than the size of the pixel block.

In one embodiment, in order to ensure that each region of the fundus image is acquired, when dividing the fundus image, the terminal determines the step size that is moved stepwise by a distance smaller than the size of the pixel block when dividing the pixel block stepwise in the fundus image.

S406, in the fundus image, division start points are determined step by step in accordance with the step size, and a plurality of pixel blocks having the size are divided step by step in accordance with the division start points.

In one embodiment, the terminal equally divides the length and width of the fundus image into a plurality of segments in accordance with the determined step size, takes the start point of each segment as the division start point of the pixel block, and gradually divides a plurality of pixel blocks having the size in accordance with the division start points.

For example, assuming that the fundus image size is 20 × 20, the size of pixel blocks to be divided is 5 × 5, and the step size is 4, the length and width of the fundus image may be divided equally into five segments, and pixel blocks of size 5 × 5 with the starting point of each segment as the division starting point, the number of pixel blocks being 25, may be divided.

In the above embodiment, the size of the pixel block to be divided and the step length of the step-by-step movement during division are determined, the division starting point is determined step by step according to the step length, and the plurality of pixel blocks with the size are divided step by step according to the division starting point.

In one embodiment, the probability tiles may be determined by a machine learning model; as shown in fig. 5, the method further includes:

s502, acquiring a fundus image sample and a corresponding reference fundus bitmap; and the reference eye fundus bitmap is used for indicating the focus category to which the pixel points at the corresponding positions of the eye fundus image sample belong.

As shown in fig. 3, if fig. 3(a) is a fundus image sample, fig. 3(b) is a corresponding reference fundus bitmap. The white pixel points in fig. 3(b) are points belonging to the focus category, that is, the corresponding region of the fundus of the user corresponding to the fundus image sample is a focus region.

In one embodiment, a method of obtaining a reference fundus bitmap includes: the terminal determines the size of the fundus image sample; acquiring focus characteristics corresponding to the fundus image; and drawing a reference fundus bitmap which accords with the size of the fundus image sample according to the focus characteristics.

In one embodiment, the terminal determines a lesion feature in the fundus image based on the input instruction, and determines a location of the lesion feature in the fundus image. And the terminal draws a reference fundus bitmap according to the size of the fundus image, and sets a color value corresponding to the focus category at a position corresponding to the reference fundus bitmap according to the determined position. Wherein the color values set by the reference fundus bitmaps of different lesion types are different.

As shown in fig. 3(a), the dark colored pixel points in the broken line frame a indicate lesion features with soft exudation, and the position of the soft exudation in the fundus image is recorded. As shown in fig. 3(B), white pixel points are drawn in a dotted line frame B corresponding to the dotted line frame a corresponding to fig. 3(a) to indicate lesion features with soft exudation. It should be noted that the black area is a background area, and the color value of the pixel point is not used for representing the lesion type.

S504, dividing the fundus image sample into a plurality of training pixel blocks.

Wherein a training pixel block is an image to be composed of a certain number of pixels in the image, such as a 256 x 256 resolution pixel block.

In one embodiment, the terminal may crop the training pixel block at a preset size on the fundus image sample. Every time one training pixel block is cut, the cutting starting point moves forwards for a certain distance, and all areas of the fundus image sample are ensured to be collected, so that a plurality of training pixel blocks are obtained. Wherein the distance of movement may be smaller than or equal to the size of the training pixel block.

For example, if the clipped training pixel block size is m × n, in order to ensure that each region of the fundus image sample is acquired, the distance of movement may be less than or equal to m when moving horizontally when dividing the fundus image sample. In the case of vertical movement, the distance of movement may be less than or equal to n.

S506, the reference fundus bitmap is divided, and a plurality of reference pixel blocks are obtained.

Wherein a reference pixel block is an image to be composed of a certain number of pixels in the image, such as a 256 x 256 resolution pixel block.

In one embodiment, the terminal may crop the reference pixel block at the reference fundus bitmap in the same size as in S504. Each time one reference pixel block is cropped, the starting point of the cropping is moved forward by a distance that ensures that each region of the reference fundus bitmap is acquired, thereby obtaining a plurality of reference pixel blocks. Wherein the distance of movement is the same as the distance of movement in S504.

For example, if the cropped reference pixel block size is m × n, in order to ensure that each region of the reference fundus bitmap is acquired, when dividing the reference fundus bitmap, the distance of movement may be less than or equal to m when moving horizontally. In the case of vertical movement, the distance of movement may be less than or equal to n.

And S508, inputting the training pixel block into a machine learning model for training to obtain a training focus area.

In one embodiment, the generating step of the machine learning model to be trained comprises: deleting a classification layer in the neural network classification model; adjusting the input size of the neural network classification model after deleting the classification layer according to the size of the pixel block to be divided; and accessing the convolution layer in the last layer of the neural network classification model with the adjusted input size to obtain the machine learning model to be trained.

In one embodiment, S508 may specifically include: the terminal inputs the training pixel blocks into a machine learning model, a plurality of training probability image blocks respectively corresponding to different focus categories are determined according to each training pixel block, the focus category to which each pixel point in the fundus image sample belongs is determined according to the training probability image blocks, and a training focus area is divided from the fundus image sample according to the focus category to which each pixel point in the fundus image sample belongs.

In one embodiment, the terminal inputs the divided training pixel blocks into a machine learning model to be trained, processes the input training pixel blocks through the machine learning model, and calculates the probability that each pixel in each training pixel block belongs to each focus category. And the terminal determines the color value of the training probability picture block according to the calculated probability, and draws the corresponding training probability picture block according to the color value to obtain a plurality of training probability picture blocks respectively corresponding to different focus categories.

The training probability blocks corresponding to the probabilities belonging to different lesion categories are different in color, that is, different in color value, for example, the training probability block belonging to soft exudation is red, and the training probability block belonging to hard exudation is yellow.

For example, assume that a fundus image is divided into m small training pixel blocks, and there are k kinds of lesion classes. The terminal inputs the m training pixel blocks into the trained machine learning model for processing, the probability that each pixel point in each training pixel block belongs to k kinds of focus categories is obtained through calculation, and k multiplied by m training probability image blocks can be obtained.

In one embodiment, the step of obtaining a training probability tile may further include: determining the characteristics corresponding to each pixel point in each training pixel block; comparing the determined characteristics with characteristics of different focus categories to obtain the probability that each pixel point in each training pixel block belongs to each focus category; determining color values of pixel points for synthesizing training probability image blocks according to the probabilities; and synthesizing pixel points with color values into a plurality of training probability image blocks according to different focus categories.

In one embodiment, after obtaining the probability that each pixel point in each training pixel block belongs to each focus category, the terminal determines whether the obtained probability is greater than or equal to a first probability threshold, and if the obtained probability is greater than or equal to the first probability threshold, it indicates that the corresponding pixel point belongs to the corresponding focus type. And determining color values of the pixel points for synthesizing the training probability image block according to the probability greater than or equal to the first probability threshold. If the color value is smaller than the first probability threshold, the corresponding color value is a colorless or black color value.

For example, assume there are k lesion classes, m1, m2 … mi … mk, respectively. The terminal respectively inputs one of the training pixel blocks into a trained machine learning model for processing, and the probability that each pixel point in the input training pixel block belongs to k focus categories is calculated to be p_m1 ⁿ、p_m2 ⁿ…p_mi ⁿ…p_mk ⁿWherein n is the number of pixels in the training pixel block, p_mi ⁿIndicating the probability that the ith pixel belongs to the category of the mi lesion. If p is_mi ⁿWhen the probability of (d) is greater than or equal to the first probability threshold, it indicates that the ith pixel belongs to the mi-th lesion class according to p_mi ⁿDetermining for synthetic trainingColor values of pixels of the probability tile.

In one embodiment, the terminal determines the lesion class to which each pixel point in the fundus image belongs according to a training probability image block.

In an embodiment, the step of determining, by the terminal, a lesion type to which each pixel point in the fundus image belongs according to the training probability segment may specifically include: the terminal splices a plurality of training probability image blocks corresponding to the same focus category to obtain training fundus bitmaps corresponding to different focus categories respectively; determining the probability that pixel points at corresponding positions in the training fundus bitmap belong to each focus category; and (4) attributing the pixel points of the corresponding positions to the focus categories of the corresponding maximum probability.

In one embodiment, after determining the probability that the pixel points at the corresponding positions in the training fundus bitmap belong to each focus category, the terminal determines whether the probability is greater than or equal to a second probability threshold, and classifies the pixel points at the corresponding positions into the focus categories corresponding to the probability greater than or equal to the second probability threshold.

In an embodiment, the step of obtaining training fundus bitmaps corresponding to different lesion categories by splicing a plurality of training probability tiles corresponding to the same lesion category by the terminal may specifically include: splicing a plurality of training probability image blocks corresponding to each focus category according to the positions of corresponding training pixel blocks divided from the fundus image; determining an overlapping area between spliced training probability images during splicing; and determining the average value of the color values of a plurality of pixel points at the same position in the overlapping region as the color value of the corresponding pixel point in the overlapping region, and obtaining the training fundus bitmap corresponding to the corresponding focus category.

When the size of the training probability image blocks is larger than the step length of the gradual movement of the training pixel blocks during division, the terminal can generate an overlapping region in the process of splicing a plurality of training probability image blocks of the same focus category. The terminal determines the overlapping area between adjacent training probability image blocks in the splicing process, and the average value of a plurality of pixel points at the same position in the overlapping area is obtained, so that the color of the overlapping area of the training probability image blocks after splicing cannot generate sudden change.

In one embodiment, the terminal receives a step instruction, the step instruction carries a step which is moved step by step when the training pixel block is divided, and the step which is moved step by step when the training pixel block is divided step by step from the fundus image is determined according to the received step instruction. Wherein the step size is less than or equal to the size of the training probability tile.

S510, adjusting parameters of the machine learning model according to differences between the pixel points in the training focus area and the pixel points at the corresponding positions in the reference pixel blocks.

In one embodiment, the terminal inputs a plurality of reference pixel blocks into the machine learning model. S510 may specifically include: the terminal determines the error between the color value of each pixel point in the training focal region and the color value of the pixel point at the corresponding position in the reference pixel block; the error is reversely propagated to each layer of the machine learning model, and the gradient of each layer parameter is obtained; and adjusting parameters of each layer in the machine learning model according to the gradient.

In one embodiment, the terminal calculates an error between a color value of each pixel point in the training focal region and a color value of a pixel point at a corresponding position in the plurality of reference pixel blocks according to the loss function. Wherein the loss function may be any of: mean Squared Error (Mean Squared Error), cross entropy Loss function, L2Loss function, and Focal Loss function.

In the above embodiment, the machine learning model is trained through a plurality of training pixel blocks divided from the fundus image sample to obtain a training focus region, parameters of the machine learning model are adjusted according to differences between pixel points of corresponding positions in each pixel point and the reference pixel block in the training focus region to obtain the machine learning model for fundus image segmentation, and the machine learning model is used to segment the fundus image to obtain a focus region for determining the focus type, so that the accuracy of fundus image segmentation is improved.

In an embodiment, as shown in fig. 6, S508 may specifically include:

s602, respectively carrying out different changes on the training pixel blocks; the variation includes at least one of a rotation process and a scaling process.

In order to improve the generalization ability of the machine learning model and improve the prediction ability of the machine learning model, the training pixel block may be subjected to rotation processing and/or scaling processing. Here, S602 is divided into the following three scenarios for explanation:

and in the scene 1, the training pixel block is subjected to rotation processing.

In one embodiment, the terminal performs random rotation processing on each training pixel block in the obtained plurality of training pixel blocks respectively. Or the terminal performs rotation processing on the plurality of training pixel blocks uniformly according to a first preset rotation angle to obtain a group of training pixel blocks; and the terminal performs rotation processing on the plurality of training pixel blocks uniformly according to other preset rotation angles different from the first preset rotation angle to obtain a plurality of groups of training pixel blocks.

Wherein, have a plurality of predetermined rotation angles, and different between a plurality of predetermined rotation angles, the angular range is 0 ~ 360.

And 2, zooming the training pixel block.

In one embodiment, the terminal performs random scaling on each training pixel block in the obtained plurality of training pixel blocks. Or the terminal performs scaling processing on the plurality of training pixel blocks uniformly according to a first preset scaling ratio to obtain a group of training pixel blocks; and the terminal performs scaling treatment on the plurality of training pixel blocks uniformly according to other preset scaling ratios different from the first preset scaling ratio to obtain a plurality of groups of training pixel blocks.

And 3, performing rotation processing and scaling processing on the training pixel block.

And performing rotation processing on the training pixel block according to the rotation mode of the scene 1, and then performing scaling processing on the rotated training pixel block according to the scaling mode of the scene 2. The specific processing steps may refer to the processing steps of scene 1 and scene 2, which are not described herein again.

And S604, carrying out image enhancement processing on the changed training pixel blocks.

In an embodiment, the implementation manner of S604 may specifically include: adjusting the brightness of the changed training pixel block, and/or adjusting the chroma of the changed training pixel block, and/or adjusting the sharpness of the changed training pixel block.

And S606, performing normalization processing on the training pixel blocks subjected to the image enhancement processing.

In one embodiment, the terminal calculates the mean and variance of the training pixel block after the image enhancement processing, and normalizes the training pixel block after the image enhancement processing according to the calculated mean and variance. The normalization processing of the training pixel block after the image enhancement processing may refer to the normalization processing of the image features in the training pixel block. The representation of the image features may be a vector or a matrix.

For example, assuming that the image feature is L, the terminal calculates the sum and variance of the image features as u and δ, respectively, and the result after the normalization process is L' ═ L/δ.

And S608, inputting the training pixel blocks subjected to the normalization processing into a machine learning model for training.

In the above embodiment, before the training pixel block is input into the machine learning model for training, the training pixel block is subjected to rotation processing, scaling processing, and image enhancement processing, so that the generalization ability of the machine learning model can be improved, and the prediction ability of the machine learning model can be improved. After the training pixel blocks are subjected to image enhancement processing, normalization processing is also carried out, so that the convergence speed can be effectively accelerated, and the training of a machine learning model can be quickly realized.

In one embodiment, as shown in fig. 7, the method may further include:

s702, the reference pixel block is changed in the same way as the training pixel block.

In order to increase the generalization ability of the machine learning model and improve the prediction ability of the machine learning model, the reference pixel block may be subjected to the same rotation processing and/or scaling processing as the reference pixel block. Here, S602 is divided into the following three scenarios for explanation:

scene 1, a rotation process is performed on the reference pixel block.

In one embodiment, the terminal performs random rotation processing on each of the obtained plurality of reference pixel blocks. Or the terminal performs rotation processing on the plurality of reference pixel blocks uniformly according to a first preset rotation angle to obtain a group of reference pixel blocks; and the terminal performs rotation processing on the plurality of reference pixel blocks uniformly according to other preset rotation angles different from the first preset rotation angle to obtain a plurality of groups of reference pixel blocks.

Scene 2, the reference pixel block is scaled.

In one embodiment, the terminal performs random scaling on each of the obtained plurality of reference pixel blocks. Or the terminal performs scaling processing on the plurality of reference pixel blocks uniformly according to a first preset scaling ratio to obtain a group of reference pixel blocks; and the terminal performs scaling treatment on the plurality of reference pixel blocks uniformly according to other preset scaling ratios different from the first preset scaling ratio to obtain a plurality of groups of reference pixel blocks.

And 3, performing rotation processing and scaling processing on the reference pixel block.

The reference pixel block is rotated in the rotation mode of scene 1 and then scaled in the scaling mode of scene 2. The specific processing steps may refer to the processing steps of scene 1 and scene 2, which are not described herein again.

In one embodiment, the terminal performs image enhancement processing on the changed reference pixel block.

In an embodiment, the step of performing, by the terminal, image enhancement processing on the changed reference pixel block may specifically include: adjusting the brightness of the changed reference pixel block, and/or adjusting the chroma of the changed reference pixel block, and/or adjusting the sharpness of the changed reference pixel block.

S510 may specifically include:

s704, inputting the changed reference pixel block into a machine learning model, and adjusting parameters of the machine learning model according to the difference between each pixel point in the training focus area and the pixel point at the corresponding position in the changed reference pixel block.

In one embodiment, the terminal inputs a plurality of image enhanced reference pixel blocks into the machine learning model. S704 may specifically include: inputting the changed reference pixel block into a machine learning model by the terminal, and determining an error between the color value of each pixel point in the training focal region and the color value of the pixel point at the corresponding position in the reference pixel block subjected to image enhancement processing; the error is reversely propagated to each layer of the machine learning model, and the gradient of each layer parameter is obtained; and adjusting parameters of each layer in the machine learning model according to the gradient.

In the above embodiment, the reference pixel block is subjected to rotation processing, scaling processing and image enhancement processing, differences between each pixel point in the lesion area and pixel points at corresponding positions in the reference pixel block subjected to the image enhancement processing are trained, and parameters of the machine learning model are adjusted, so that generalization capability of the machine learning model can be improved, and prediction capability of the machine learning model can be improved.

In the conventional scheme, focus segmentation models of fundus images are mainly as follows: segmentation based on artificially defined features and lesion segmentation based on deep learning. However, the method of segmenting based on artificially defined features has poor robustness; deep learning based methods require extensive training data. The existing method based on deep learning can not output accurate regional contour by dividing the fundus image into small blocks to classify the focus.

In order to solve the above problem, an embodiment of the present invention provides a fundus image segmentation method, including:

(1) and (4) preparing data.

A fundus image sample of high resolution and a lesion region corresponding to the fundus image sample are prepared. The lesion region is labeled with a bitmap called a reference fundus bitmap, and as shown in fig. 3(b), the value of each pixel in the bitmap indicates to which lesion type the pixel at the corresponding position of the fundus image sample belongs. The corresponding lesion category in the lesion area may include, among others, normal area, hard effusion, microaneurysms, hemorrhage, and soft effusion.

(2) Model building, as shown in fig. 8, the method of model building includes:

s802, designing a machine learning model backbone, wherein the model backbone is a neural network classification model which is obtained by removing a classification module and adjusting an input size.

The neural network classification model may include: a deep convolutional neural network model, or a deep full-lap machine network model, or other deep neural network models. The deep convolutional neural network model may be a ResNet101 network model. The deep full-volume machine network model may be a U-Net network model. Other deep neural network models may be, for example, Incepton-Resnet-V2, ResNext, NasNet, MobileNet, etc.

S804, one convolution layer is connected to the last layer of the model trunk, the input of the convolution layer is the output of the last layer of the model trunk, and the dimension of the output of the convolution layer is the lesion category number.

(3) Model training, as shown in fig. 9, the model training method includes:

randomly initializing parameters of the machine learning model or importing network parameters of Pre-train on other data sets into the machine learning model. A subset of the training set is recursively sampled to update the model parameters, and each iteration performs data enhancement on the data in the subset:

s902, randomly cutting a training pixel block with 256 multiplied by 256 resolution from each fundus image sample, and carrying out the same operation on the corresponding reference fundus bitmap.

And S904, carrying out random rotation processing and/or random scaling processing between 0 and 360 degrees on the training pixel block, and carrying out the same operation on the corresponding reference fundus bitmap.

And S906, randomly adjusting the brightness, the chroma and the definition of the training pixel block.

And S908, performing normalization processing on the training pixel block.

S910, inputting the training pixel block subjected to the normalization processing into a machine learning model for forward calculation.

And S912, inputting the reference pixel block cut according to the reference fundus bitmap into a machine learning model, and calculating the error of the classification result of each pixel in the training pixel block according to the loss function.

The Loss function may be a pixel-by-pixel cross entropy Loss function, an L2Loss function, a Focal Loss function, or the like.

And S914, reversely propagating the calculated error to the machine learning model, and calculating the gradient of the model parameter.

S916, updating the model parameters based on the gradient.

(4) A prediction stage, as shown in fig. 10, the fundus image segmentation method in the prediction stage includes:

and S1002, acquiring fundus images of the patient, and inputting the fundus images into the trained machine learning model.

S1004, in the fundus image, one 256 × 256 pixel block is clipped for the image every 224 pixels in width and height.

And S1006, inputting the pixel blocks obtained by cutting into a machine learning model, and calculating to obtain the probability that each pixel in the pixel blocks belongs to a specific focus category to obtain a probability image block.

For example, assuming that there are m pixel blocks, the m pixel blocks are input into the machine learning model, respectively, so that k × m 256 × 256 probability tiles can be obtained. Wherein k is the number of lesion categories.

And S1008, splicing probability image blocks of the pixel blocks to obtain k fundus bitmaps for dividing the focus. In this case, the overlapping portions can be obtained by averaging.

S1010, extracting the pixel with the maximum probability at the same position of the k fundus bitmaps, and taking a bitmap formed by the extracted pixels as a prediction result of the final focus.

By implementing the embodiment, the following beneficial effects can be achieved:

1) and (3) establishing a machine learning model by using a deep learning correlation technique to obtain an accurate segmentation result.

2) The application method is simple, the judgment speed is high, and the focus area on the image can be automatically identified only by inputting the fundus image by a user.

Fig. 2 is a flowchart illustrating a fundus image segmentation method in one embodiment. It should be understood that, although the steps in the flowchart of fig. 2 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 2 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

As shown in fig. 11, in one embodiment, there is provided a fundus image segmentation apparatus specifically including: a fundus image acquisition module 1102, a first pixel block division module 1104, a probability tile determination module 1106, a lesion category determination module 1108, and a segmentation module 1110; wherein:

a fundus image acquisition module 1102 for acquiring a fundus image to be segmented;

a first pixel block division module 1104 for dividing a plurality of pixel blocks from the fundus image;

a probability block determination module 1106, configured to determine, according to each of the pixel blocks, a plurality of probability blocks respectively corresponding to different lesion categories; each color value in the probability picture block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category;

a lesion category determining module 1108, configured to determine, according to the probability image block, a lesion category to which each pixel point in the fundus image belongs;

a segmentation module 1110, configured to segment a lesion region from the fundus image according to a lesion type to which each pixel of the fundus image belongs.

In one embodiment, the first pixel block division module 1104 is further configured to determine a size of a pixel block to be divided; determining a step size to be moved step by step when pixel blocks are divided step by step from the fundus image; the step size is smaller than the size of the pixel block; in the fundus image, division start points are determined step by step in accordance with the step size, and a plurality of pixel blocks of the size are divided step by step in accordance with the division start points.

In one embodiment, the probabilistic tile determination module 1106 is further configured to determine corresponding features for each pixel point in each pixel block; comparing the determined characteristics with characteristics of different focus categories to obtain the probability that each pixel point in each pixel block belongs to each focus category; determining color values of pixel points for synthesizing probability image blocks according to the probability; and synthesizing pixel points with color values into a plurality of probability image blocks according to different focus categories.

In the above embodiment, the determined features are compared with the features of different lesion categories to obtain the probability that each pixel point in each pixel block belongs to each lesion category, thereby determining the lesion category. The color values of the pixel points used for synthesizing the probability image blocks are determined according to the probability, the pixel points with the color values are synthesized into a plurality of probability image blocks according to different focus categories, the visualization of the focus is realized, and medical personnel can judge the focus categories through fundus bitmaps composed of the probability image blocks.

In one embodiment, the lesion category determination module 1108 is further configured to splice a plurality of probability patches corresponding to the same lesion category to obtain fundus bitmaps corresponding to different lesion categories, respectively; determining the probability that pixel points at corresponding positions in the fundus bitmap belong to each focus category; and (4) attributing the pixel points of the corresponding positions to the focus categories of the corresponding maximum probability.

In the above embodiment, the probability that the pixel points at the corresponding positions in the fundus bitmap belong to each lesion category is determined, and the pixel points at each corresponding position are assigned to the lesion category with the corresponding maximum probability, so that the final lesion category is judged, and the lesion category is judged in an automatic manner.

In one embodiment, the lesion category determining module 1108 is further configured to splice a plurality of probability tiles corresponding to each lesion category according to the positions of the corresponding pixel blocks divided from the fundus image; determining an overlapping area between spliced probability images during splicing; and determining the average value of the color values of a plurality of pixel points at the same position in the overlapping region as the color value of the corresponding pixel point in the overlapping region, and obtaining the fundus bitmap corresponding to the corresponding focus category.

In the above embodiment, the average value of the color values of the plurality of pixel points at the same position in the overlapping region is determined as the color value of the corresponding pixel point in the overlapping region, so that the influence on the judgment of the lesion category in the region when a plurality of pixel blocks in the overlapping region are overlapped is avoided, and the accuracy of judging the lesion category can be further improved.

In one embodiment, as shown in fig. 12, the apparatus further comprises: a reference fundus bitmap acquisition module 1112, a training pixel block partitioning module 1114, a second pixel block partitioning module 1116, a training module 1118, and a parameter adjustment module 1120; wherein:

a reference fundus bitmap acquisition module 1112 for acquiring a fundus image sample and a corresponding reference fundus bitmap; the reference eye fundus bitmap is used for indicating the focus category to which the pixel points at the corresponding positions of the eye fundus image samples belong;

a training pixel block dividing module 1114 for dividing the fundus image sample into a plurality of training pixel blocks;

a second pixel block division module 1116 for dividing the reference fundus bitmap to obtain a plurality of reference pixel blocks;

a training module 1118, configured to input the training pixel blocks into a machine learning model for training, so as to obtain a training focus area;

the parameter adjusting module 1120 is configured to adjust parameters of the machine learning model according to differences between the pixel points in the training focal region and the pixel points at the corresponding positions in the reference pixel block.

In the above embodiment, the machine learning model is trained through a plurality of training pixel blocks divided from the fundus image sample to obtain a training focus region, parameters of the machine learning model are adjusted according to differences between pixel points of corresponding positions in each pixel point and the reference pixel block in the training focus region to obtain the machine learning model for fundus image segmentation, and the machine learning model is used to segment the fundus image to obtain a focus region for determining focus categories, so that accuracy of fundus image segmentation is improved.

In one embodiment, the training module 1118 is further configured to make different changes to the training pixel blocks, respectively; the change includes at least one of a rotation process and a scaling process; carrying out image enhancement processing on the changed training pixel blocks; carrying out normalization processing on the training pixel blocks subjected to image enhancement processing; and inputting the training pixel blocks subjected to the normalization processing into a machine learning model for training.

In one embodiment, as shown in fig. 12, the apparatus further comprises: a processing module 1122; wherein:

a processing module 1122 for performing the same changes on the reference pixel block as the training pixel block;

the parameter adjusting module 1120 is further configured to input the changed reference pixel block into the machine learning model, and adjust parameters of the machine learning model according to differences between each pixel point in the training focal region and a pixel point at a corresponding position in the changed reference pixel block.

In one embodiment, as shown in fig. 12, the apparatus further comprises: a deletion module 1124, a size adjustment module 1126, and an access module 1128; wherein:

a deleting module 1124 for deleting the classification layer in the neural network classification model;

a size adjusting module 1126, configured to adjust an input size of the neural network classification model after the classification layer is deleted according to the size of the pixel block to be divided;

and the access module 1128 is configured to access the convolutional layer in the last layer of the neural network classification model with the adjusted input size, so as to obtain a machine learning model to be trained.

In the above embodiment, the model for fundus image segmentation can be obtained by processing the machine learning model, which is beneficial to improving the accuracy and efficiency of fundus image segmentation.

In one embodiment, as shown in fig. 12, the apparatus further comprises: a size determination module 1130, a lesion feature acquisition 1132, and a reference fundus bitmap drawing module 1134; wherein:

a size determination module 1130 for determining the size of the fundus image sample;

a lesion feature acquisition 1132, configured to acquire a lesion feature corresponding to the fundus image;

and a reference fundus bitmap drawing module 1134, configured to draw a reference fundus bitmap that conforms to the size of the fundus image sample according to the lesion feature.

In the embodiment, the reference fundus bitmap which accords with the size of the fundus image sample is drawn, and the reference fundus bitmap is used as training output, so that the parameter adjustment of the model is more accurate when the machine learning model is trained.

In one embodiment, the parameter adjustment module 1120 is further configured to determine an error between a color value of each pixel point in the training focal region and a color value of a pixel point at a corresponding position in the plurality of reference pixel blocks; the error is reversely propagated to each layer of the machine learning model, and the gradient of each layer parameter is obtained; and adjusting parameters of each layer in the machine learning model according to the gradient.

In the embodiment, the gradient is calculated through error back propagation, and the parameters of each layer in the machine learning model are adjusted according to the gradient, so that the learning efficiency can be improved, and the training speed is accelerated.

FIG. 13 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may specifically be the terminal 110 in fig. 1. As shown in fig. 13, the computer apparatus includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer apparatus stores an operating system, and may further store a computer program that, when executed by the processor, causes the processor to implement the fundus image segmentation method. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to execute a fundus image segmentation method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

Those skilled in the art will appreciate that the architecture shown in fig. 13 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, the fundus image segmentation apparatus provided herein may be implemented in the form of a computer program executable on a computer device as shown in fig. 13. The memory of the computer device may store therein various program modules constituting the fundus image segmentation apparatus, such as a fundus image acquisition module 1102, a first pixel block division module 1104, a probability patch determination module 1106, a lesion class determination module 1108, and a segmentation module 1110 shown in fig. 11. The computer program constituted by the respective program modules causes the processor to execute the steps in the fundus image segmentation method according to each embodiment of the present application described in the present specification.

For example, the computer device shown in fig. 13 may execute S202 by the fundus image acquisition module 1102 in the fundus image segmentation apparatus shown in fig. 11. The computer device may perform S204 by the first pixel block division module 1104. The computer device may perform S206 by the probabilistic tile determination module 1106. The computer device may perform S208 by the lesion classification determination module 1108. The computer device may perform S210 through the segmentation module 1110.

In one embodiment, there is provided a computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of: acquiring a fundus image to be segmented; dividing a plurality of pixel blocks from a fundus image; determining a plurality of probability image blocks respectively corresponding to different focus categories according to each pixel block; each color value in the probability picture block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category; determining the focus category of each pixel point in the fundus image according to the probability picture block; and segmenting a focus area from the fundus image according to the focus category to which each pixel point of the fundus image belongs.

In one embodiment, the computer program, when executed by the processor, causes the processor to specifically perform the steps of: determining the size of a pixel block to be divided; determining a step size to be moved step by step when pixel blocks are divided step by step from the fundus image; the step size is smaller than the size of the pixel block; in the fundus image, division start points are determined step by step in accordance with the step size, and a plurality of pixel blocks of the size are divided step by step in accordance with the division start points.

In one embodiment, the computer program, when executed by the processor, causes the processor to perform the steps of determining, from each block of pixels, a plurality of probability tiles respectively corresponding to different lesion classes, in particular: determining the corresponding characteristics of each pixel point in each pixel block; comparing the determined characteristics with characteristics of different focus categories to obtain the probability that each pixel point in each pixel block belongs to each focus category; determining color values of pixel points for synthesizing probability image blocks according to the probability; and synthesizing pixel points with color values into a plurality of probability image blocks according to different focus categories.

In one embodiment, the computer program, when executed by the processor, causes the processor to perform the steps of determining a lesion type to which each pixel point in the fundus image belongs based on the probability map, in particular: splicing a plurality of probability image blocks corresponding to the same focus category to obtain fundus bitmaps corresponding to different focus categories respectively; determining the probability that pixel points at corresponding positions in the fundus bitmap belong to each focus category; and (4) attributing the pixel points of the corresponding positions to the focus categories of the corresponding maximum probability.

In one embodiment, the computer program, when executed by the processor, causes the processor to perform the step of concatenating a plurality of probability patches corresponding to the same lesion class to obtain fundus bitmaps corresponding to different lesion classes, in particular, the step of: splicing a plurality of probability image blocks corresponding to each focus category according to the positions of the corresponding pixel blocks divided from the fundus image; determining an overlapping area between spliced probability images during splicing; and determining the average value of the color values of a plurality of pixel points at the same position in the overlapping region as the color value of the corresponding pixel point in the overlapping region, and obtaining the fundus bitmap corresponding to the corresponding focus category.

In one embodiment, the probability tiles are determined by a machine learning model; the computer program, when executed by the processor, causes the processor to further perform the steps of: acquiring a fundus image sample and a corresponding reference fundus bitmap; the reference eye fundus bitmap is used for indicating the focus category to which the pixel points at the corresponding positions of the eye fundus image samples belong; dividing the fundus image sample into a plurality of training pixel blocks; dividing a reference fundus bitmap to obtain a plurality of reference pixel blocks; inputting the training pixel block into a machine learning model for training to obtain a training focus area; and adjusting parameters of the machine learning model according to the difference between each pixel point in the training focus area and the pixel point at the corresponding position in the reference pixel block.

In one embodiment, the computer program, when executed by the processor, causes the processor to perform the steps of inputting training pixel blocks into machine learning model training, in particular: respectively carrying out different changes on the training pixel blocks; the change includes at least one of a rotation process and a scaling process; carrying out image enhancement processing on the changed training pixel blocks; carrying out normalization processing on the training pixel blocks subjected to image enhancement processing; and inputting the training pixel blocks subjected to the normalization processing into a machine learning model for training.

In one embodiment, the computer program, when executed by the processor, causes the processor to further perform the steps of: changing the reference pixel block in the same way as the training pixel block; the step of adjusting parameters of the machine learning model according to the difference between each pixel point in the training focal region and the pixel point at the corresponding position in the reference pixel block specifically comprises: and inputting the changed reference pixel block into a machine learning model, and adjusting parameters of the machine learning model according to the difference between each pixel point in the training focus area and the pixel point at the corresponding position in the changed reference pixel block.

In one embodiment, the computer program, when executed by the processor, causes the processor to further perform the steps of: generating a machine learning model to be trained, wherein the generating step of the machine learning model to be trained comprises: deleting a classification layer in the neural network classification model; adjusting the input size of the neural network classification model after deleting the classification layer according to the size of the pixel block to be divided; and accessing the convolution layer in the last layer of the neural network classification model with the adjusted input size to obtain the machine learning model to be trained.

In one embodiment, the computer program, when executed by the processor, causes the processor to specifically perform the steps of:

in one embodiment, the computer program, when executed by the processor, causes the processor to further perform the steps of: determining the size of a fundus image sample; acquiring focus characteristics corresponding to the fundus image; and drawing a reference fundus bitmap which accords with the size of the fundus image sample according to the focus characteristics.

In one embodiment, when the computer program is executed by the processor to adjust the parameters of the machine learning model according to the difference between each pixel point in the training focal region and the pixel point at the corresponding position in the reference pixel block, the processor is specifically caused to execute the following steps: determining an error between the color value of each pixel point in the training focal region and the color value of the pixel point at the corresponding position in the plurality of reference pixel blocks; the error is reversely propagated to each layer of the machine learning model, and the gradient of each layer parameter is obtained; and adjusting parameters of each layer in the machine learning model according to the gradient.

In one embodiment, a computer readable storage medium is provided, storing a computer program that, when executed by a processor, causes the processor to perform the steps of: acquiring a fundus image to be segmented; dividing a plurality of pixel blocks from a fundus image; determining a plurality of probability image blocks respectively corresponding to different focus categories according to each pixel block; each color value in the probability picture block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category; determining the focus category of each pixel point in the fundus image according to the probability picture block; and segmenting a focus area from the fundus image according to the focus category to which each pixel point of the fundus image belongs.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A fundus image segmentation apparatus, comprising:

the pixel block dividing module is used for cutting the fundus image according to a preset size to obtain a plurality of pixel blocks;

the probability image block determining module is used for processing each input pixel block through a machine learning model and calculating the probability that each pixel point in each pixel block belongs to the corresponding focus category; determining a color value according to the calculated probability, and drawing corresponding probability picture blocks according to the color value to obtain a plurality of probability picture blocks respectively corresponding to different focus categories; each color value in the probability picture block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category;

and the segmentation module is used for extracting pixel points which belong to the same position and have the highest probability from fundus bitmaps which are obtained by splicing the probability image blocks and respectively correspond to the focus categories, and combining the extracted pixel points to obtain a focus area.

2. The apparatus of claim 1, wherein the pixel block partitioning module is further configured to determine a size of a pixel block to be partitioned; determining a step size to be moved step by step when pixel blocks are divided step by step from the fundus image; the step size is smaller than the size of the pixel block; in the fundus image, division start points are determined step by step in accordance with the step size, and a plurality of pixel blocks of the size are divided step by step in accordance with the division start points.

3. The apparatus of claim 1, wherein the probabilistic tile determination module is further configured to:

determining the corresponding characteristics of each pixel point in each pixel block;

comparing the determined characteristics with characteristics of different focus categories to obtain the probability that each pixel point in each pixel block belongs to each focus category;

determining color values of pixel points for synthesizing probability image blocks according to the probability;

and synthesizing the pixel points with the color values into a plurality of probability image blocks according to different focus categories.

4. The apparatus of claim 1, wherein the lesion classification determination module is further configured to:

splicing a plurality of probability image blocks corresponding to the same focus category to obtain fundus bitmaps corresponding to different focus categories respectively;

determining the probability that pixel points at corresponding positions in the fundus bitmap belong to each focus category;

and (4) attributing the pixel points of the corresponding positions to the focus categories of the corresponding maximum probability.

5. The apparatus of claim 4, wherein the lesion category determination module is further configured to:

splicing a plurality of probability image blocks corresponding to each focus category according to the positions of the corresponding pixel blocks divided from the fundus image;

determining an overlapping area between spliced probability images during splicing;

and determining the average value of the color values of a plurality of pixel points at the same position in the overlapping region as the color value of the corresponding pixel point in the overlapping region, and obtaining the fundus bitmap corresponding to the corresponding focus category.

6. The apparatus of any of claims 1 to 5, wherein the probability patches are determined by a machine learning model; the device further comprises:

the reference fundus bitmap acquisition module is used for acquiring a fundus image sample and a corresponding reference fundus bitmap; pixel points in the reference eye fundus bitmap are reference labels and are used for representing the focus category to which the pixel points at the corresponding positions of the eye fundus image sample belong;

the training pixel block dividing module is used for dividing the fundus image sample into a plurality of training pixel blocks;

a second pixel block division module for dividing the reference fundus bitmap to obtain a plurality of reference pixel blocks;

the training module is used for inputting the training pixel blocks into a machine learning model for training to obtain a training focus area;

and the parameter adjusting module is used for adjusting parameters of the machine learning model according to the difference between each pixel point in the training focal region and the pixel point at the corresponding position in the reference pixel block.

7. The apparatus of claim 6, wherein the training module is further configured to:

respectively carrying out different changes on the training pixel blocks; the varying includes at least one of a rotation process and a scaling process;

carrying out image enhancement processing on the changed training pixel blocks;

carrying out normalization processing on the training pixel blocks subjected to image enhancement processing;

and inputting the training pixel blocks subjected to the normalization processing into a machine learning model for training.

8. The apparatus of claim 7, further comprising:

a processing module for performing the same change on the reference pixel block as the training pixel block;

and the parameter adjusting module is also used for inputting the changed reference pixel block into a machine learning model and adjusting the parameters of the machine learning model according to the difference between each pixel point in the training focus area and the pixel point at the corresponding position in the changed reference pixel block.

9. The apparatus of claim 6, further comprising:

the deleting module is used for deleting a classification layer in the neural network classification model;

the size adjusting module is used for adjusting the input size of the neural network classification model after the classification layer is deleted according to the size of the pixel block to be divided;

and the access module is used for accessing the convolution layer in the last layer of the neural network classification model with the adjusted input size to obtain the machine learning model to be trained.

10. The apparatus of claim 6, further comprising:

a size determination module for determining the size of the fundus image sample;

acquiring focus characteristics, namely acquiring focus characteristics corresponding to the fundus image;

and the reference fundus bitmap drawing module is used for drawing a reference fundus bitmap which accords with the size of the fundus image sample according to the focus characteristics.

11. The apparatus of claim 6, wherein the parameter adjustment module is further configured to:

determining an error between the color value of each pixel point in the training focal region and the color value of the pixel point at the corresponding position in the plurality of reference pixel blocks;

propagating the error back to each layer of the machine learning model to obtain a gradient for each layer parameter;

and adjusting parameters of each layer in the machine learning model according to the gradient.

12. A storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of:

acquiring a fundus image to be segmented;

cutting the fundus image according to a preset size to obtain a plurality of pixel blocks;

processing each input pixel block through a machine learning model, and calculating the probability that each pixel point in each pixel block belongs to the corresponding focus category; determining a color value according to the calculated probability, and drawing corresponding probability picture blocks according to the color value to obtain a plurality of probability picture blocks respectively corresponding to different focus categories; each color value in the probability picture block represents the probability that the corresponding pixel point in the pixel block belongs to each focus category;

and extracting pixel points which belong to the same position and have the highest probability from fundus bitmaps which are obtained by splicing the probability image blocks and respectively correspond to the focus categories, and combining the extracted pixel points to obtain a focus area.

13. The storage medium according to claim 12, wherein the computer program, when executed by the processor, causes the processor to specifically execute the step of cropping the fundus image by a preset size to obtain a plurality of pixel blocks:

determining the size of a pixel block to be divided;

determining a step size to be moved step by step when pixel blocks are divided step by step from the fundus image; the step size is smaller than the size of the pixel block;

in the fundus image, division start points are determined step by step in accordance with the step size, and a plurality of pixel blocks of the size are divided step by step in accordance with the division start points.

14. The storage medium according to claim 12, wherein the computer program, when executed by the processor, causes the processor to perform the steps of determining, from each of the pixel blocks, a plurality of probability tiles respectively corresponding to different lesion classes, in particular:

15. The storage medium according to claim 12, wherein the computer program, when executed by the processor, causes the processor to specifically execute the steps of determining a lesion class package to which each pixel point in the fundus image belongs based on the probability map:

16. The storage medium of claim 15, wherein the computer program, when executed by the processor, causes the processor to perform the step of stitching a plurality of probability patches corresponding to a same lesion class to obtain fundus bitmaps corresponding to different lesion classes, respectively, to cause the processor to perform the steps of:

17. The storage medium of any one of claims 12 to 16, wherein the probability patches are determined by a machine learning model; the computer program, when executed by the processor, causes the processor to further perform the steps of:

acquiring a fundus image sample and a corresponding reference fundus bitmap; pixel points in the reference eye fundus bitmap are reference labels and are used for representing the focus category to which the pixel points at the corresponding positions of the eye fundus image sample belong;

dividing the fundus image sample into a plurality of training pixel blocks;

dividing the reference fundus bitmap to obtain a plurality of reference pixel blocks;

inputting the training pixel block into a machine learning model for training to obtain a training focus area;

and adjusting parameters of the machine learning model according to the difference between each pixel point in the training focal region and the pixel point at the corresponding position in the reference pixel block.

18. The storage medium of claim 17, wherein the computer program, when executed by the processor to perform the step of inputting the training pixel block into machine learning model training, causes the processor to perform the steps of:

carrying out image enhancement processing on the changed training pixel blocks;

19. The storage medium of claim 18, wherein the computer program, when executed by the processor, causes the processor to further perform the steps of:

performing the same changes to the reference pixel block as the training pixel block;

and inputting the changed reference pixel block into a machine learning model, and adjusting parameters of the machine learning model according to the difference between each pixel point in the training focal region and the pixel point at the corresponding position in the changed reference pixel block.

20. The storage medium of claim 17, wherein the computer program, when executed by the processor, causes the processor to further perform the steps of:

deleting a classification layer in the neural network classification model;

adjusting the input size of the neural network classification model after deleting the classification layer according to the size of the pixel block to be divided;

and accessing the convolution layer at the last layer of the neural network classification model with the adjusted input size to obtain the machine learning model to be trained.

21. The storage medium of claim 17, wherein the computer program, when executed by the processor, causes the processor to further perform the steps of:

determining the size of a fundus image sample;

acquiring focus characteristics corresponding to the fundus image;

and drawing a reference fundus bitmap which accords with the size of the fundus image sample according to the focus characteristics.

22. The storage medium of claim 17, wherein the computer program, when executed by the processor, causes the processor to perform the step of adjusting parameters of the machine learning model based on differences between pixels in the training focal region and pixels in corresponding locations in the reference pixel block by:

23. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to:

acquiring a fundus image to be segmented;

24. The computer device according to claim 23, wherein the computer program, when executed by the processor, causes the processor to specifically execute the step of cropping the fundus image to a preset size, resulting in a plurality of pixel blocks:

determining the size of a pixel block to be divided;

25. A computer device according to claim 23 wherein the computer program, when executed by the processor, causes the processor to perform the steps of determining from each of the pixel blocks a plurality of probability tile packets respectively corresponding to different lesion classes, in particular:

26. The computer device according to claim 23, wherein the computer program, when executed by the processor, causes the processor to perform the steps of determining a lesion class package to which each pixel point in the fundus image belongs from the probability patch, in particular:

27. The computer device according to claim 26, wherein the computer program, when executed by the processor, causes the processor to perform the step of stitching a plurality of probability tiles corresponding to the same lesion class to obtain fundus bitmaps corresponding to different lesion classes, respectively, to cause the processor to perform in particular the steps of:

28. The computer device of any one of claims 23 to 27, wherein the probability patches are determined by a machine learning model; the computer program, when executed by the processor, causes the processor to further perform the steps of:

dividing the fundus image sample into a plurality of training pixel blocks;

29. The computer device according to claim 28, wherein the computer program, when being executed by the processor for entering the training pixel block into a machine learning model training, causes the processor to carry out in particular the steps of:

carrying out image enhancement processing on the changed training pixel blocks;

30. The computer device of claim 29, wherein the computer program, when executed by the processor, causes the processor to further perform the steps of:

31. The computer device according to claim 28, wherein the computer program, when executed by the processor, causes the processor to further perform the steps of:

deleting a classification layer in the neural network classification model;

32. The computer device according to claim 28, wherein the computer program, when executed by the processor, causes the processor to further perform the steps of:

determining the size of a fundus image sample;

acquiring focus characteristics corresponding to the fundus image;

33. The computer device of claim 28, wherein the computer program, when executed by the processor, causes the processor to perform the step of adjusting parameters of the machine learning model based on differences between pixels in the training focal region and pixels in corresponding positions in the reference pixel block, by: