CN111461165A

CN111461165A - Image recognition method, recognition model training method, related device and equipment

Info

Publication number: CN111461165A
Application number: CN202010121559.5A
Authority: CN
Inventors: 杨爽; 李嘉辉; 黄晓迪
Original assignee: Shanghai Sensetime Intelligent Technology Co Ltd
Current assignee: Shanghai Sensetime Intelligent Technology Co Ltd
Priority date: 2020-02-26
Filing date: 2020-02-26
Publication date: 2020-07-28
Also published as: TW202133043A; WO2021169161A1; JP2022537781A; TWI767506B; KR20210110823A

Abstract

The application discloses an image recognition method, a training method of a recognition model, a related device and equipment, wherein the image recognition method comprises the following steps: acquiring a pathological image to be identified; carrying out target detection on the pathological image to be recognized by adopting a detection sub-model in the recognition model to obtain a detection area containing target cells in the pathological image to be recognized; and carrying out first classification processing on the detection area by utilizing a classification submodel in the recognition model to obtain the category of the target cell. According to the scheme, the target cells in the pathological image can be accurately and efficiently identified.

Description

Image recognition method, recognition model training method, related device and equipment

Technical Field

The present application relates to the field of artificial intelligence technologies, and in particular, to an image recognition method, a recognition model training method, and related devices and apparatuses.

Background

With the development of artificial intelligence technologies such as neural networks and deep learning, the neural network model is trained, and the trained neural network model is utilized to meet related business requirements in the medical field, so that the neural network model is gradually favored by people.

In the related business requirements, because the domestic cytopathology doctors are seriously deficient, the method utilizes the artificial intelligence technology to carry out auxiliary identification on the pathological images so as to screen target cells such as pathological cells, and has important significance under the condition of deficient current cytopathology medical resources. In view of the above, how to accurately and efficiently identify target cells in a pathological image is an urgent problem to be solved.

Disclosure of Invention

The application provides an image recognition method, a recognition model training method, a related device and equipment.

A first aspect of the present application provides an image recognition method, including: acquiring a pathological image to be identified; carrying out target detection on the pathological image to be recognized by adopting a detection sub-model in the recognition model to obtain a detection area containing target cells in the pathological image to be recognized; and carrying out first classification processing on the detection area by utilizing a classification submodel in the recognition model to obtain the category of the target cell.

Therefore, the target detection is carried out on the obtained pathological image to be recognized by adopting the detection submodel in the recognition model, so that a detection area containing target cells in the pathological image to be recognized is obtained, then the analysis submodel in the recognition model is used for overhauling the first classification treatment on the detection area, so that the classification of the target cells is obtained, the target cells can be firstly detected, then the target cells are classified, the detection and the classification are separated, and the target cells in the pathological image can be accurately and efficiently recognized.

The method comprises the following steps of performing target detection on a pathological image to be recognized by adopting a detection submodel in a recognition model, and obtaining a detection area containing target cells in the pathological image to be recognized, wherein the detection area comprises: performing second classification processing on the pathological image to be recognized by utilizing the first part of the detection sub-model to obtain an image classification result of the pathological image to be recognized, wherein the image classification result is used for indicating whether the pathological image to be recognized contains target cells; and if the image classification result shows that the pathological image to be recognized contains the target cells, performing region detection on the pathological image to be recognized by using the second part of the detection sub-model to obtain a detection region containing the target cells.

Therefore, the second classification processing is carried out on the pathological image to be recognized through the first part of the detection sub-model, the image classification result of the pathological image to be recognized is obtained, the image classification result is used for indicating whether the pathological image to be recognized contains the target cells or not, when the image classification result indicates that the pathological image to be recognized contains the target cells, the second part of the detection sub-model is used for carrying out region detection on the pathological image to be recognized, the detection region containing the target cells is obtained, therefore, the dynamic detection of the target cells can be realized, and the efficiency of target cell recognition is improved.

After the second classification processing is performed on the pathological image to be recognized by using the first part of the detection sub-model to obtain the image classification result of the pathological image to be recognized, the method further comprises the following steps: if the image classification result shows that the pathological image to be recognized does not contain the target cell, the first part outputs a detection result prompt that the image to be recognized does not contain the target cell.

Therefore, when the image classification result shows that the pathological image to be identified does not contain the target cell, the first part outputs the detection result prompt that the image to be identified does not contain the target cell, so that the dynamic detection of the target cell can be realized, and the efficiency of target cell identification is improved.

The method comprises the following steps of performing target detection on a pathological image to be recognized by adopting a detection submodel in the recognition model, and obtaining a detection area containing target cells in the pathological image to be recognized, wherein the detection area further comprises the following steps: and performing feature extraction on the pathological image to be identified by using the third part of the detection sub-model to obtain the image features of the pathological image to be identified.

Therefore, the third part of the detection sub-model is used for carrying out feature extraction on the pathological image to be identified to obtain the image features of the pathological image to be identified, so that the pathological image to be identified can be firstly carried out, and then the detection sub-model is subsequently utilized to carry out other processing on the basis, and therefore the operation efficiency of the model can be favorably improved.

The method for carrying out the second classification processing on the pathological image to be recognized by utilizing the first part of the detection sub-model to obtain the image classification result of the pathological image to be recognized comprises the following steps: and carrying out second classification processing on the image characteristics by utilizing the first part of the detection sub-model to obtain an image classification result of the pathological image to be identified.

Therefore, the first part of the detection sub-model is used for carrying out second classification processing on the image features extracted from the third part to obtain the image classification result of the pathological image to be identified, and the accuracy of the classification processing can be improved.

The method for detecting the pathological image to be identified by using the second part of the detection sub-model comprises the following steps of: and carrying out region detection on the image characteristics by using a second part of the detection submodel to obtain a detection region containing the target cells.

Therefore, the second part of the detection submodel is used for carrying out region detection on the image features to obtain the detection region containing the target cells, and the accuracy of target cell identification can be improved.

The first part is a global classification network, the second part is an image detection network, and the third part is a feature extraction network; the feature extraction network comprises at least one of a deformable convolution layer and a global information enhancement module.

Therefore, the accuracy of identifying the polymorphic target cells can be improved by setting the feature extraction network to include the deformable convolution layer, and the feature extraction network can be favorable for acquiring long-distance features with dependency relationship and improving the accuracy of identifying the target cells by setting the feature extraction network to include at least one of the global information enhancement modules.

The method for obtaining the category of the target cell by carrying out first classification processing on the detection area by utilizing a classification submodel in the recognition model comprises the following steps: carrying out feature extraction on a detection area of the pathological image to be identified by utilizing the classification submodel to obtain image features of the detection area; and carrying out first classification processing on the image characteristics of the detection area to obtain the category of the target cell.

Therefore, the method can be used for extracting the characteristics of the detection area of the pathological image to be identified to obtain the image characteristics of the detection area, and performing the first classification processing on the image characteristics of the detection area to obtain the category of the target cell, thereby being beneficial to improving the efficiency of the classification processing.

The target cell includes either a single diseased cell or a diseased cell cluster, and the category of the target cell is used to indicate the degree of disease of the target cell.

Therefore, the target cell includes either a single diseased cell or a diseased cell cluster, and it is possible to facilitate identification of the single diseased cell or the diseased cell cluster, and the category of the target cell is used to indicate the degree of disease of the target cell, which is advantageous for achieving disease classification of the target cell.

The second aspect of the present application provides a training method for a recognition model, where the recognition model includes a detection submodel and a classification submodel, and the training method includes: acquiring a first sample image and a second sample image, wherein the first sample image is marked with an actual area corresponding to a target cell, and the second sample image is marked with an actual category of the target cell; carrying out target detection on the first sample image by using the detection sub-model to obtain a prediction region containing target cells in the first sample image, and carrying out first classification processing on the second sample image by using the classification sub-model to obtain the prediction category of the target cells; determining a first loss value of the detection submodel based on the actual region and the prediction region, and determining a second loss value of the classification submodel based on the actual category and the prediction category; and correspondingly adjusting the parameters of the detection submodel and the classification submodel by using the first loss value and the second loss value.

Therefore, in the training process, the target cells can be detected firstly, then the target cells are classified, and the detection and classification are separated, so that the problem of unbalanced sample data categories can be solved, the accuracy of the model obtained by training can be improved, and the accuracy and the efficiency of target cell identification can be improved.

The method for detecting the target of the first sample image by using the detection submodel to obtain the prediction region containing the target cell in the first sample image comprises the following steps: performing second classification processing on the first sample image to obtain an image classification result of the first sample image, wherein the image classification result is used for indicating whether the first sample image contains target cells or not; and if the image classification result shows that the first sample image contains the target cell, performing region detection on the first sample image to obtain a prediction region containing the target cell.

Therefore, in the training process, when the image classification result shows that the first sample image contains the target cell, the first sample image is subjected to region detection to obtain a prediction region containing the target cell, so that the capability of the model for identifying the positive and negative samples can be enhanced, the false detection probability is reduced, the accuracy of the model obtained by training can be improved, and the accuracy of target cell identification can be improved.

Before the detecting sub-model is used to perform target detection on the first sample image to obtain a prediction region containing target cells in the first sample image, and the classifying sub-model is used to perform first classification processing on the second sample image to obtain a prediction category of the target cells, the method further comprises: performing data enhancement on the first sample image and the second sample image; and/or performing normalization processing on pixel values in the first sample image and the second sample image; the target cell includes either a single diseased cell or a cluster of diseased cells, and the category of the target cell is used to indicate the degree of disease of the target cell.

Therefore, the sample diversity can be improved by performing data enhancement on the first sample image and the second sample image, so that overfitting is avoided, and the generalization performance of the model is improved; the pixel values in the first sample image and the second sample image are normalized, so that the convergence rate of the model can be improved; the target cell includes either a single diseased cell or a diseased cell cluster, the category of the target cell is used to indicate the degree of disease of the target cell, and is advantageous for identifying the single diseased cell and the diseased cell cluster, and the category of the target cell is used to indicate the degree of disease of the target cell, and is advantageous for achieving the disease classification of the target cell.

A third aspect of the present application provides an image recognition apparatus comprising: the system comprises an image acquisition module, an image detection module and an image classification module, wherein the image acquisition module is used for acquiring pathological images to be identified; the image detection module is used for carrying out target detection on the pathological image to be identified by adopting a detection sub-model in the identification model to obtain a detection area containing target cells in the pathological image to be identified; the image classification module is used for carrying out first classification processing on the detection area by utilizing a classification sub-model in the identification model to obtain the category of the target cell.

The fourth aspect of the present application provides a training apparatus for recognition models, where the recognition models include a detection submodel and a classification submodel, and the training apparatus for recognition models includes: the device comprises an image acquisition module, a model execution module, a loss determination module and a parameter adjustment module, wherein the image acquisition module is used for acquiring a first sample image and a second sample image, the first sample image is marked with an actual area corresponding to a target cell, and the second sample image is marked with an actual category of the target cell; the model execution module is used for carrying out target detection on the first sample image by using the detection sub-model to obtain a prediction region containing target cells in the first sample image, and carrying out first classification processing on the second sample image by using the classification sub-model to obtain the prediction category of the target cells; the loss determining module is used for determining a first loss value of the detection submodel based on the actual region and the prediction region, and determining a second loss value of the classification submodel based on the actual category and the prediction category; and the parameter adjusting module is used for correspondingly adjusting the parameters of the detection submodel and the classification submodel by utilizing the first loss value and the second loss value.

A fifth aspect of the present application provides an electronic device, which includes a memory and a processor coupled to each other, wherein the processor is configured to execute program instructions stored in the memory to implement the image recognition method in the first aspect or implement the training method of the recognition model in the second aspect.

A sixth aspect of the present application provides a computer-readable storage medium on which program instructions are stored, which program instructions, when executed by a processor, implement the image recognition method in the first aspect, or implement the training method of the recognition model in the second aspect described above.

According to the scheme, the detection submodel in the identification model is adopted to carry out target detection on the obtained pathological image to be identified, so that a detection area containing target cells in the pathological image to be identified is obtained, then the analysis submodel in the identification model is used for overhauling the first classification treatment on the detection area, the classification of the target cells is obtained, then the target cells can be detected firstly, then the target cells are classified, the detection and classification are separated, and therefore the target cells in the pathological image can be identified accurately and efficiently.

Drawings

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of an image recognition method of the present application;

FIG. 2 is a schematic diagram of an embodiment of an image recognition method according to the present application;

FIG. 3 is a schematic flow chart diagram illustrating another embodiment of an image recognition method according to the present application;

FIG. 4 is a state embodiment of another embodiment of the image recognition method of the present application;

FIG. 5 is a schematic flow chart diagram illustrating an embodiment of a training method for recognition models according to the present application;

FIG. 6 is a block diagram of an embodiment of an image recognition apparatus according to the present application;

FIG. 7 is a block diagram of an embodiment of a training apparatus for recognition models according to the present application;

FIG. 8 is a block diagram of an embodiment of an electronic device of the present application;

FIG. 9 is a block diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed Description

The following describes in detail the embodiments of the present application with reference to the drawings attached hereto.

In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, interfaces, techniques, etc. in order to provide a thorough understanding of the present application.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship. Further, the term "plurality" herein means two or more than two.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of an image recognition method according to the present application. Specifically, the method may include the steps of:

step S11: and acquiring a pathological image to be identified.

The pathology image to be identified may include, but is not limited to: cervical pathology images, liver pathology images, and kidney pathology images, but are not limited thereto.

Step S12: and performing target detection on the pathological image to be recognized by adopting a detection sub-model in the recognition model to obtain a detection area containing target cells in the pathological image to be recognized.

The recognition model includes a detection sub-model, in a specific implementation scenario, the detection sub-model may employ a Fast RCNN (Region with relational Neural Networks) network model, in another specific implementation scenario, the detection sub-model may also employ Fast RCNN, YO L O (You Only L ook one), and the like, which is not limited herein.

Detecting the pathological image to be identified by using the detection sub-model to obtain a detection area containing target cells in the pathological image to be identified, for example, detecting the cervical pathological image to obtain a detection area containing squamous epithelial cells in the cervical pathological cells; or, the pathological liver image is detected to obtain a detection area containing the pathological cells in the pathological liver image, and when the pathological image to be identified is another image, the process can be analogized, and the examples are not repeated. In an implementation scenario, the detection region may specifically be represented by a central coordinate of a rectangle containing the target cell and a length and a width of the rectangle, for example, (50,60,10,20) may represent a rectangle with a length of 10 and a width of 20 centered on the pixel coordinate (50,60) in the pathology image to be identified, and may also be represented by a ratio of the central coordinate of the rectangle containing the target cell and the length and the width of the rectangle to a predetermined rectangle, respectively, for example, the predetermined rectangle may be a rectangle with a length of 10 and a width of 20, and (50,60,1,1) may represent a rectangle with a length of 10 and a width of 20 centered on the pixel coordinate (50,60) in the pathology image to be identified, which is not limited herein.

In a specific implementation scenario, the pathological image to be recognized may also be an image that does not include target cells, and at this time, the detection submodel in the recognition model is used to perform target detection on the pathological image to be recognized, and since the detection area is not obtained, a prompt that the pathological image to be recognized does not include the target cells can be output, so that the step of subsequent classification processing is omitted, and the operation efficiency of the model is improved. For example, a prompt that the cervical pathology image does not contain squamous epithelial cells may be directly output, and other pathology images may be analogized, which is not to be taken as an example.

In a specific implementation scenario, please refer to fig. 2 in combination, and fig. 2 is a schematic state diagram of an embodiment of the image recognition method according to the present application. As shown in fig. 2, the pathological image to be identified is a cervical pathological image, and the pathological image to be identified performs target detection through a detection sub-model in the identification model, so as to obtain two detection regions including target cells.

Step S13: and carrying out first classification processing on the detection area by utilizing a classification submodel in the recognition model to obtain the category of the target cell.

The recognition model may further include a classification sub-model, and in a specific implementation scenario, the classification sub-model may employ an EfficientNet network model. In another specific implementation scenario, the classification submodel may also use ResNet, MobileNet, etc., which are not limited herein.

The classification submodel in the recognition model is used for classifying the detection area, so that the category of the target cell can be obtained, specifically, in order to improve the classification efficiency, the classification submodel can be used for extracting the features of the detection area of the pathological image to be recognized to obtain the image features of the detection area, and therefore the first classification processing is performed on the image features of the detection area to obtain the category of the target cell. For example, pooling and full-link processing may be performed on the image features of the detection region to obtain the category of the target cell, which is not described herein again.

In one implementation scenario, the target cells may specifically include, but are not limited to, the following categories, taking the lesion image to be identified as a cervical pathology image, high-grade squamous cell intraepithelial neoplasia (L SI L), mild-grade squamous cell intraepithelial neoplasia (HSI L), atypical squamous cell with unknown significance (ASC-US), atypical squamous cell with no exclusion of high-grade intraepithelial neoplasia (ASC-H), when the pathology image to be identified is another pathology image, the same may be so, and no further examples are given here.

In a specific implementation scenario, referring to fig. 2 in a continuous manner, the classification submodel classifies two detection areas detected by the detection submodel to obtain categories of target cells contained in the two detection areas, wherein the target cells in one detection area are mild squamous cell intraepithelial neoplasia (HSI L), and the target cells in the other detection area are atypical squamous cells (ASC-H) which cannot exclude high intraepithelial neoplasia.

In another specific implementation scenario, the classification submodel may further perform a first classification on the detection region to obtain a class of the target cell and a confidence thereof, where the confidence indicates that the true class of the target cell is the confidence of the class predicted by the model, and the confidence is higher, please continue to refer to fig. 2, the classification submodel performs a classification on the detection region to obtain a class of the target cell and a confidence thereof, respectively, where the target cell in one detection region is a mild squamous cell intraepithelial neoplasia (HSI L) and has a confidence of 0.97 (i.e., a confidence of 97%), and the target cell in another detection region is an atypical squamous cell (ASC-H) that cannot exclude high intraepithelial neoplasia and has a confidence of 0.98 (i.e., a confidence of 98%).

Referring to fig. 3, fig. 3 is a schematic flowchart illustrating an image recognition method according to another embodiment of the present application. Specifically, the method may include the steps of:

step S31: and acquiring a pathological image to be identified.

Please refer to the related steps in the previous embodiment.

Step S32: and classifying the pathological images to be recognized by utilizing the first part of the detection sub-model to obtain the image classification results of the pathological images to be recognized.

The image classification result is used to indicate whether the pathological image to be identified contains the target cell, specifically, "0" may be used to indicate that the pathological image to be identified does not contain the target cell, and "1" may be used to indicate that the pathological image to be identified contains the target cell, which is not limited herein.

In an implementation scenario, the first part of the detection sub-model is a global classification network, the global classification network is a neural network model including neurons, and the global classification network is different from the classification sub-model in the foregoing embodiment, and is configured to perform two classification processes on the pathological image to be identified to obtain an image classification result of whether the pathological image to be identified includes the target cell. In a specific implementation scenario, in order to distinguish from the classification process of the classification submodel, the classification process of the first part of the detection submodel may be referred to as a second classification process, which is not limited herein.

Step S33: and judging whether the image classification result indicates that the pathological image to be identified contains the target cell, if so, executing step S34, otherwise, executing step S36.

And judging whether the pathological image to be recognized contains the target cell or not through an image classification result, if so, carrying out next processing on the pathological image to be recognized, otherwise, carrying out next processing on the pathological image to be recognized is not needed, so that the classification processing of whether the pathological image to be recognized contains the target cell is separated from a detection area for specifically detecting the target cell, the operation efficiency of the model can be further improved, and the efficiency of recognizing the target cell in the image is further improved.

Step S34: and carrying out region detection on the pathological image to be identified by utilizing the second part of the detection sub-model to obtain a detection region containing the target cells.

In an implementation scenario, the second part of the detection submodel is an image detection network, the image detection network is a neural network model including neurons, for example, the detection submodel adopts fast RCNN, the second part may be an rpn (region pro social networks) network, and when the detection submodel is another network model, the same can be done, and so on, which is not illustrated here.

In one implementation scenario, to improve the accuracy of target cell identification. The third part of the detection sub-model may also be used to perform feature extraction on the pathological image to be identified to obtain the image features of the pathological image to be identified, specifically, the third part may be a feature extraction network, and in a specific implementation scenario, the feature extraction network may be a ResNet101 network, or the feature extraction network may also be a ResNet50 network, and the like, which is not limited herein. In another specific implementation scenario, in order to improve the accuracy of identifying the multi-morphological target cell, the feature extraction network may include a deformable convolution layer (deformable convolution), and the deformable convolution is further adjusted by displacement based on the position information adopted for the space, so as to implement feature extraction for cells with different morphologies. In a further specific implementation scenario, in order to obtain long-distance dependent features, thereby improving the accuracy of target cell identification, the feature extraction network may further include a global information enhancement module. Please refer to fig. 4, fig. 4 is a state diagram of another embodiment of the image recognition method of the present application, after feature extraction is performed on a pathological image to be recognized, a first part of a detection sub-model may be used to classify image features to obtain an image classification result of the pathological image to be recognized, and when the image classification result indicates that the pathological image to be recognized includes target cells (i.e., when the image classification result is positive), a second part of the detection sub-model is used to perform region detection on the image features to obtain a detection region including the target cells, so as to perform subsequent classification processing.

Step S35: and classifying the detection area by using a classification submodel in the recognition model to obtain the category of the target cell.

Please refer to the related steps in the previous embodiment.

Step S36: the first part outputs a detection result prompt that the target cell is not included in the image to be recognized.

When the image detection result indicates that the pathological image to be recognized does not contain the target cell (namely, the image classification result is negative), the next processing is not needed, so that the detection result prompt (namely, the prompt with the negative result) that the image to be recognized does not contain the target cell can be directly output, the operation efficiency of the model is improved, and the efficiency of recognizing the target cell in the image is improved.

Different from the foregoing embodiment, the second classification processing is performed on the pathological image to be recognized through the first part of the detection sub-model to obtain the image classification result of the pathological image to be recognized, the image classification result is used for indicating whether the pathological image to be recognized contains the target cell, and when the image classification result indicates that the pathological image to be recognized contains the target cell, the second part of the detection sub-model is used for performing the region detection on the pathological image to be recognized to obtain the detection region containing the target cell, so that the dynamic detection of the target cell can be realized, and the efficiency of target cell identification can be improved.

Referring to fig. 5, fig. 5 is a schematic flowchart of an embodiment of a training method for a recognition model in the present application, in the embodiment of the present application, the recognition model may specifically include a detection submodel and a classification submodel, and specifically, the method may include the following steps:

step S51: a first sample image and a second sample image are acquired.

In an embodiment of the present application, the first sample image is labeled with an actual region corresponding to the target cell, the actual region may be represented by a center coordinate of a rectangle containing the target cell and a length and a width of the rectangle, for example, (50,60,10,20) may be represented by a rectangle which is 10 long and 20 wide and is centered on a pixel point (50,60) in the first sample image, the second sample image is labeled with an actual category of the target cell, and in one embodiment, the actual category of the target cell is used to represent a lesion degree of the target cell.

In one implementation scenario, the first and second sample images are pathological images, which may include, but are not limited to: cervical pathology images, liver pathology images, kidney pathology images. Taking the first sample image and the second sample image as cervical pathology images as an example, the target cell may be a squamous epithelial cell. When the first sample image and the second sample image are other pathological images, the analogy can be repeated, and no one example is given here.

In an implementation scene, data enhancement can be performed on the acquired ground sample image and the second sample image, so that sample diversity is improved, overfitting is avoided, and generalization performance of the model is improved. In one particular implementation scenario, data enhancement may be performed using operations including, but not limited to: random rotation, random inversion, color perturbation, gamma correction, gaussian noise.

In an implementation scenario, the pixel values in the first sample image and the second sample image may also be normalized, so as to improve the convergence rate of the model. In a specific implementation scenario, a first mean value and a first variance of pixel values of all first sample images may be counted first, and then the first mean value is subtracted from the pixel value of each first sample image, and then the first mean value is divided by the first variance, so as to perform normalization processing on each first sample image; and a second mean value and a second variance of the pixel values of all the second sample images can be counted, the second mean value is subtracted from the pixel value of each second sample image, and the second mean value is divided by the second variance, so that each second yann image is subjected to normalization processing.

Step S52: and carrying out target detection on the first sample image by using the detection sub-model to obtain a prediction region containing target cells in the first sample image, and carrying out first classification processing on the second sample image by using the classification sub-model to obtain the prediction category of the target cells.

The detection submodel may adopt fast RCNN, and reference may be made to the relevant steps in the foregoing embodiments, which are not described herein again. The prediction region may be represented by a central coordinate of a rectangle and a length and width of the rectangle, for example, (70,80,10,20) may be used to represent a prediction region located in the first sample image and having the pixel point (70,80) as a center, the length of 10 and the width of 20, and the prediction region may be represented by a ratio of the central coordinate of a rectangle and the length and width of the rectangle to a length and width of a preset rectangle, respectively, for example, a preset rectangle may be set, and the length of the preset rectangle is 10 and the width of 20, and (70,80,1,1) may be used to represent a prediction region located in the first sample image and having the (70,80) as an image center, the length of 10 and the width of 20. The classification sub-model may adopt an EfficientNet network model, and reference may be made to the relevant steps in the foregoing embodiments, which are not described herein again.

In an implementation scenario, in order to improve the capability of the model to identify positive and negative samples and implement dynamic prediction to improve the operation efficiency of the model, in the process of performing target detection on a first sample image by using a detection sub-model to obtain a prediction region of the first sample image containing target cells, a second classification process may be performed on the first sample image to obtain an image classification result of the first sample image, where the image classification result is used to indicate whether the first sample image contains target cells, and if the image classification result indicates that the first sample image contains target cells, the first sample image is subjected to region detection to obtain the prediction region containing target cells. In addition, the detection sub-model may further include a first part and a second part, where the first part is configured to perform classification processing on the first sample image to obtain an image classification result whether the first sample image includes the target cell, and the second part is configured to perform region detection on the first sample image when the first sample image includes the target cell to obtain a prediction region including the target cell. In addition, the detection submodel may further include a third portion, configured to perform feature extraction on the first sample image to obtain image features of the first sample image, so that the first portion performs feature extraction on the image features to obtain an image classification result of the first sample image, and the second portion performs region detection on the image features to obtain a prediction region including the target cell. Specifically, the first part may be a global classification network, the second part is an image detection network, and the third part is a feature extraction network, where the feature extraction network includes at least one of a deformable convolution layer and a global information enhancement module, and reference may be made to the relevant steps in the foregoing embodiments, which are not described herein again.

Step S53: a first loss value of the detection submodel is determined based on the actual region and the prediction region, and a second loss value of the classification submodel is determined based on the actual category and the prediction category.

In a specific implementation scenario, a mean square error loss function, a cross entropy loss function, and the like may be used to determine a first loss value of the detection submodel. In another specific implementation scenario, the cross entropy loss function may be used to determine the second loss value of the classification submodel, which is not described herein again.

Step S54: and correspondingly adjusting the parameters of the detection submodel and the classification submodel by using the first loss value and the second loss value.

Specifically, the parameters of the detection submodel and the classification submodel may be adjusted by using a random gradient descent optimization method, an exponential average weighting optimization method, an Adam optimization method, and the like, which are not described herein again.

In addition, the first sample image and the second sample image may be divided into a plurality of small batches (batch), and the detection sub-model and the classification sub-model may be trained in a mini-batch training mode. In one implementation scenario, a training end condition may also be set, and when the training end condition is satisfied, the training may be ended. Specifically, the training end condition may include, but is not limited to: the number of iterations of the training is greater than or equal to a preset threshold (e.g., 100, 500, etc.); the first loss value and the second loss value are smaller than a preset loss threshold value and are not reduced any more; the model performance obtained by verifying the detection submodel and the classification submodel by using a verification data set is not improved any more, and is not limited herein.

According to the scheme, in the training process, the target cells can be detected firstly, then the target cells are classified, and the detection and classification are separated, so that the problem of unbalanced sample data categories can be solved, the accuracy of a model obtained by training can be improved, and the accuracy and the efficiency of target cell identification can be improved.

Referring to fig. 6, fig. 6 is a schematic diagram of a framework of an embodiment of an image recognition apparatus 60 according to the present application. The image recognition device 60 comprises an image acquisition module 61, an image detection module 62 and an image classification module 63, wherein the image acquisition module 61 is used for acquiring pathological images to be recognized; the image detection module 62 is configured to perform target detection on the pathological image to be identified by using a detection sub-model in the identification model, so as to obtain a detection area containing target cells in the pathological image to be identified; the image classification module 63 is configured to perform a first classification process on the detection region by using a classification submodel in the recognition model to obtain a category of the target cell.

According to the scheme, the detection submodel in the identification model is adopted to carry out target detection on the obtained pathological image to be identified, so that the detection area containing the target cells in the pathological image to be identified is obtained, the analysis submodel in the identification model is used for overhauling the first classification treatment on the detection area, the category of the target cells is obtained, the target cells can be firstly detected, then the target cells are classified, the detection and classification are separated, and the target cells in the pathological image can be accurately and efficiently identified

In some embodiments, the image detection module 62 includes a first sub-module configured to perform a second classification process on the pathological image to be identified by using a first part of the detection sub-model to obtain an image classification result of the pathological image to be identified, where the image classification result is used to indicate whether the pathological image to be identified includes the target cell, and the image detection module 62 further includes a second sub-module configured to perform a region detection on the pathological image to be identified by using a second part of the detection sub-model when the image classification result indicates that the pathological image to be identified includes the target cell, so as to obtain a detection region including the target cell.

In some embodiments, the image detection module 62 further includes a result prompting sub-module, configured to output a detection result prompt that the target cell is not included in the image to be recognized when the image classification result indicates that the target cell is not included in the pathological image to be recognized.

Different from the foregoing embodiment, after performing the second classification processing on the pathological image to be identified by using the first part of the detection sub-model to obtain the image classification result of the pathological image to be identified, the method further includes: if the image classification result shows that the pathological image to be recognized does not contain the target cell, the first part outputs a detection result prompt that the image to be recognized does not contain the target cell.

In some embodiments, the image detection module 62 further includes a third part sub-module, configured to perform feature extraction on the pathological image to be identified by using a third part of the detection sub-model, so as to obtain image features of the pathological image to be identified.

Different from the embodiment, the third part of the detection sub-model is used for carrying out feature extraction on the pathological image to be identified to obtain the image features of the pathological image to be identified, so that the pathological image to be identified can be firstly carried out, and then the detection sub-model is subsequently utilized to carry out other processing on the basis, and therefore the operation efficiency of the model can be favorably improved.

In some embodiments, the first part sub-module is specifically configured to perform second classification processing on the image features by using the first part of the detection sub-model, so as to obtain an image classification result of the pathological image to be identified.

Different from the foregoing embodiment, the first part of the detection sub-model is used to perform the second classification processing on the image features extracted from the third part, so as to obtain the image classification result of the pathological image to be identified, and the accuracy of the classification processing can be improved.

In some embodiments, the second part sub-module is specifically configured to perform region detection on the image feature using the second part of the detection sub-model, so as to obtain a detection region containing the target cell.

Different from the foregoing embodiment, the second part of the detection submodel is used to perform region detection on the image features to obtain a detection region containing the target cells, which can be beneficial to improving the accuracy of target cell identification.

In some embodiments, the first part is a global classification network, the second part is an image detection network, and the third part is a feature extraction network; the feature extraction network comprises at least one of a deformable convolution layer and a global information enhancement module.

Different from the foregoing embodiment, the accuracy of identifying the polymorphic target cells can be improved by setting the feature extraction network to include the deformable convolution layer, and the accuracy of identifying the target cells can be improved by setting the feature extraction network to include at least one of the global information enhancement modules, which is beneficial to obtaining long-distance features with dependency relationship.

In some embodiments, the image classification module 63 includes a feature extraction sub-module, configured to perform feature extraction on a detection region of the pathological image to be identified by using the classification sub-model to obtain an image feature of the detection region, and the image classification module 63 includes a classification processing sub-module, configured to perform first classification processing on the image feature of the detection region to obtain a category of the target cell.

Different from the foregoing embodiment, the method and the device for classifying target cells obtain image features of the detection region by performing feature extraction on the detection region of the pathological image to be recognized, and perform first classification processing on the image features of the detection region to obtain a category of the target cells, which can be beneficial to improving the efficiency of classification processing.

In some embodiments, the target cell includes any one of a single diseased cell, a cluster of diseased cells, and the category of the target cell is used to indicate the degree of disease of the target cell.

Unlike the foregoing embodiment, the target cell includes any one of a single diseased cell and a diseased cell cluster, which can facilitate identification of the single diseased cell and the diseased cell cluster, and the category of the target cell is used to indicate the degree of disease of the target cell, which facilitates achieving the disease classification of the target cell.

Referring to fig. 7, fig. 7 is a block diagram illustrating an embodiment of a training apparatus 70 for recognizing models according to the present application. The recognition model comprises a detection submodel and a classification submodel, the training device 70 of the recognition model comprises an image acquisition module 71, a model execution module 72, a loss determination module 73 and a parameter adjustment module 74, the image acquisition module 71 is used for acquiring a first sample image and a second sample image, wherein the first sample image is marked with an actual area corresponding to a target cell, and the second sample image is marked with an actual category of the target cell; the model execution module 72 is configured to perform target detection on the first sample image by using the detection sub-model to obtain a prediction region containing target cells in the first sample image, and perform first classification processing on the second sample image by using the classification sub-model to obtain a prediction category of the target cells; the loss determining module 73 is configured to determine a first loss value of the detection submodel based on the actual region and the prediction region, and determine a second loss value of the classification submodel based on the actual category and the prediction category; the parameter adjusting module 74 is configured to adjust parameters of the detection submodel and the classification submodel according to the first loss value and the second loss value.

In some embodiments, the model executing module 72 includes an initial classification sub-module, configured to perform a second classification process on the first sample image to obtain an image classification result of the first sample image, where the image classification result is used to indicate whether the first sample image includes the target cell, and the model executing module 72 includes a region detecting sub-module, configured to perform region detection on the first sample image to obtain a prediction region including the target cell when the image classification result indicates that the first sample image includes the target cell.

Different from the embodiment, in the training process, when the image classification result indicates that the first sample image contains the target cell, the first sample image is subjected to region detection to obtain a prediction region containing the target cell, so that the capability of the model for identifying positive and negative samples can be enhanced, the false detection probability is reduced, the accuracy of the model obtained by training can be improved, and the accuracy of target cell identification can be improved.

In some embodiments, the training device 70 for identifying a model further comprises a data enhancement module for performing data enhancement on the first sample image and the second sample image.

Different from the embodiment, the sample diversity can be improved by performing data enhancement on the first sample image and the second sample image, so that overfitting is avoided, and the generalization performance of the model is improved.

In some embodiments, the training device 70 for identifying a model further includes a normalization processing module for normalizing the pixel values in the first sample image and the second sample image.

Unlike the foregoing embodiment, by performing normalization processing on the pixel values in the first sample image and the second sample image, it is possible to advantageously increase the convergence speed of the model.

Unlike the foregoing embodiment, the target cell includes any one of a single diseased cell and a diseased cell cluster, the category of the target cell is used to indicate the degree of disease of the target cell, which can facilitate identification of the single diseased cell and the diseased cell cluster, and the category of the target cell is used to indicate the degree of disease of the target cell, which facilitates achievement of a disease classification of the target cell.

Referring to fig. 8, fig. 8 is a schematic block diagram of an embodiment of an electronic device 80 according to the present application. The electronic device 80 comprises a memory 81 and a processor 82 coupled to each other, and the processor 82 is configured to execute program instructions stored in the memory 81 to implement the steps of any of the embodiments of the image recognition method described above, or to implement the steps of any of the embodiments of the training method of the recognition model described above. In one particular implementation scenario, the electronic device 80 may include, but is not limited to: a microcomputer, a server, and the electronic device 80 may also include a mobile device such as a notebook computer, a tablet computer, and the like, which is not limited herein.

In particular, the processor 82 is configured to control itself and the memory 81 to implement the steps of any of the above-described embodiments of the image recognition method, or to implement the steps of any of the above-described embodiments of the training method of the recognition model. The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The processor 82 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 82 may be collectively implemented by an integrated circuit chip.

According to the scheme, the target cells in the pathological image can be accurately and efficiently identified.

Referring to fig. 9, fig. 9 is a block diagram illustrating an embodiment of a computer-readable storage medium 90 according to the present application. The computer readable storage medium 90 stores program instructions 901 capable of being executed by a processor, the program instructions 901 being configured to implement the steps of any of the above-described embodiments of the image recognition method, or the steps of any of the above-described embodiments of the training method of the recognition model.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely one type of logical division, and an actual implementation may have another division, for example, a unit or a component may be combined or integrated with another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some interfaces, and may be in an electrical, mechanical or other form.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on network elements. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Claims

1. An image recognition method, comprising:

acquiring a pathological image to be identified;

carrying out target detection on the pathological image to be recognized by adopting a detection sub-model in a recognition model to obtain a detection area containing target cells in the pathological image to be recognized;

and carrying out first classification treatment on the detection area by utilizing a classification submodel in the identification model to obtain the category of the target cell.

2. The image recognition method according to claim 1, wherein the performing target detection on the pathological image to be recognized by using a detection submodel in the recognition model to obtain a detection region containing target cells in the pathological image to be recognized comprises:

performing second classification processing on the pathological image to be recognized by using the first part of the detection sub-model to obtain an image classification result of the pathological image to be recognized, wherein the image classification result is used for indicating whether the pathological image to be recognized contains the target cell;

and if the image classification result shows that the pathological image to be identified contains the target cells, performing region detection on the pathological image to be identified by using the second part of the detection sub-model to obtain a detection region containing the target cells.

3. The image recognition method according to claim 2, wherein after the performing a second classification process on the pathological image to be recognized by using the first part of the detection sub-model to obtain an image classification result of the pathological image to be recognized, the method further comprises:

if the image classification result indicates that the pathological image to be recognized does not contain the target cell, the first part outputs a detection result prompt that the image to be recognized does not contain the target cell.

4. The image recognition method according to claim 2 or 3, wherein the performing target detection on the pathological image to be recognized by using a detection submodel in the recognition model to obtain a detection region containing target cells in the pathological image to be recognized further comprises:

and performing feature extraction on the pathological image to be identified by using the third part of the detection sub-model to obtain the image features of the pathological image to be identified.

5. The image recognition method according to claim 4, wherein the performing a second classification process on the pathological image to be recognized by using the first part of the detection sub-model to obtain an image classification result of the pathological image to be recognized comprises:

and carrying out second classification processing on the image characteristics by utilizing the first part of the detection sub-model to obtain an image classification result of the pathological image to be identified.

6. The image recognition method according to claim 4, wherein the performing region detection on the pathology image to be recognized by using the second part of the detection submodel to obtain a detection region including the target cell comprises:

and carrying out region detection on the image characteristics by using the second part of the detection submodel to obtain a detection region containing the target cells.

7. The image recognition method according to any one of claims 4 to 6, wherein the first part is a global classification network, the second part is an image detection network, and the third part is a feature extraction network; wherein the feature extraction network comprises at least one of a deformable convolutional layer and a global information enhancement module.

8. The image recognition method according to claim 1 or 2, wherein the performing a first classification process on the detection region by using a classification submodel in the recognition model to obtain the class of the target cell comprises:

performing feature extraction on the detection area of the pathological image to be identified by using the classification submodel to obtain image features of the detection area;

and carrying out first classification processing on the image characteristics of the detection area to obtain the category of the target cell.

9. The image recognition method according to any one of claims 1 to 8, wherein the target cell includes any one of a single diseased cell and a diseased cell cluster, and the category of the target cell is used to indicate a degree of a disease of the target cell.

10. A training method for a recognition model, wherein the recognition model includes a detection submodel and a classification submodel, the method comprising:

acquiring a first sample image and a second sample image, wherein the first sample image is marked with an actual region corresponding to a target cell, and the second sample image is marked with an actual category of the target cell;

performing target detection on the first sample image by using the detection sub-model to obtain a prediction region containing target cells in the first sample image, and performing first classification processing on the second sample image by using the classification sub-model to obtain a prediction category of the target cells;

determining a first loss value of the detection submodel based on the actual region and the prediction region, and determining a second loss value of the classification submodel based on the actual category and the prediction category;

and correspondingly adjusting the parameters of the detection submodel and the classification submodel by using the first loss value and the second loss value.

11. A training method as claimed in claim 10, wherein the performing target detection on the first sample image by using the detection submodel to obtain a predicted region of the first sample image containing target cells comprises:

performing second classification processing on the first sample image to obtain an image classification result of the first sample image, wherein the image classification result is used for indicating whether the first sample image contains the target cell;

and if the image classification result shows that the first sample image contains the target cell, performing region detection on the first sample image to obtain a prediction region containing the target cell.

12. A training method as claimed in claim 10 or 11, wherein before the performing target detection on the first sample image by using the detection submodel to obtain a predicted region of the first sample image including target cells, and performing first classification processing on the second sample image by using the classification submodel to obtain a predicted classification of the target cells, the method further comprises:

performing data enhancement on the first sample image and the second sample image;

and/or performing normalization processing on pixel values in the first sample image and the second sample image;

the target cell includes any one of a single lesion cell and a cluster of lesion cells, and the category of the target cell is used to indicate the degree of lesion of the target cell.

13. An image recognition apparatus, comprising:

the image acquisition module is used for acquiring a pathological image to be identified;

the image detection module is used for carrying out target detection on the pathological image to be identified by adopting a detection sub-model in the identification model to obtain a detection area containing target cells in the pathological image to be identified;

and the image classification module is used for carrying out first classification processing on the detection area by utilizing a classification sub-model in the identification model to obtain the category of the target cell.

14. A training apparatus for a recognition model, wherein the recognition model includes a detection submodel and a classification submodel, the training apparatus for the recognition model includes:

the image acquisition module is used for acquiring a first sample image and a second sample image, wherein the first sample image is marked with an actual area corresponding to the target cell, and the second sample image is marked with an actual category of the target cell;

the model execution module is used for carrying out target detection on the first sample image by using the detection sub-model to obtain a prediction region containing target cells in the first sample image, and carrying out first classification processing on the second sample image by using the classification sub-model to obtain a prediction type of the target cells;

a loss determining module, configured to determine a first loss value of the detection submodel based on the actual region and the prediction region, and determine a second loss value of the classification submodel based on the actual category and the prediction category;

and the parameter adjusting module is used for correspondingly adjusting the parameters of the detection submodel and the classification submodel by utilizing the first loss value and the second loss value.

15. An electronic device comprising a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the image recognition method of any one of claims 1 to 9 or the training method of the recognition model of any one of claims 10 to 12.

16. A computer-readable storage medium, on which program instructions are stored, which program instructions, when executed by a processor, implement the image recognition method of any one of claims 1 to 9, or the training method of the recognition model of any one of claims 10 to 12.