CN113222989A - Image grading method and device, storage medium and electronic equipment - Google Patents
Image grading method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN113222989A CN113222989A CN202110641027.9A CN202110641027A CN113222989A CN 113222989 A CN113222989 A CN 113222989A CN 202110641027 A CN202110641027 A CN 202110641027A CN 113222989 A CN113222989 A CN 113222989A
- Authority
- CN
- China
- Prior art keywords
- image
- target part
- region
- target
- interest
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000003709 image segmentation Methods 0.000 claims abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000000605 extraction Methods 0.000 claims description 68
- 238000001514 detection method Methods 0.000 claims description 40
- 238000013145 classification model Methods 0.000 claims description 12
- 230000003902 lesion Effects 0.000 description 14
- 230000037311 normal skin Effects 0.000 description 8
- 230000011218 segmentation Effects 0.000 description 8
- 238000013528 artificial neural network Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000013135 deep learning Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 210000002615 epidermis Anatomy 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 238000002591 computed tomography Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 206010003011 Appendicitis Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 244000000010 microbial pathogen Species 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000002600 positron emission tomography Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 210000003491 skin Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20104—Interactive definition of region of interest [ROI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention discloses an image grading method, an image grading device, a storage medium and electronic equipment. The method comprises the following steps: acquiring an image to be processed, and extracting a target part image in the image to be processed; carrying out image segmentation of an interested region in the target part image to obtain a mask image of the interested region; and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image. According to the technical scheme, when image classification is carried out, the characteristics of the target part image and the mask image of the interested region can be effectively learned by means of the twin network model, so that the similarity of the target part image and the mask image of the interested region is measured, the grade of the target part image is accurately judged based on the similarity, automatic classification is realized, and the image processing efficiency is improved.
Description
Technical Field
Embodiments of the present invention relate to image processing technologies, and in particular, to an image classification method and apparatus, a storage medium, and an electronic device.
Background
A lesion refers to a confined, diseased tissue with pathogenic microorganisms, called a lesion.
The focus identification and focus area grading technology generally adopts methods of manual calculation and judgment and interactive segmentation detection on an input picture to identify the area of a focus in the picture, calculate the area of the focus, classify the focus, make corresponding classification reference and treatment scheme recommendation strategies, and the like.
However, the judgment by adopting a manual calculation mode has complex and troublesome operation and also consumes longer measurement time; the interactive segmentation detection mode cannot realize automatic segmentation, needs manual intervention, occupies large memory and has low operation speed.
Disclosure of Invention
The embodiment of the invention provides an image grading method, an image grading device, a storage medium and electronic equipment, which are used for automatically and accurately judging the grade of a target part image and improving the image processing efficiency.
In a first aspect, an embodiment of the present invention provides an image classification method, including:
acquiring an image to be processed, and extracting a target part image in the image to be processed;
carrying out image segmentation of an interested region in the target part image to obtain a mask image of the interested region;
and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image.
In a second aspect, an embodiment of the present invention further provides an image classification apparatus, including:
the target extraction module is used for acquiring an image to be processed and extracting a target part image in the image to be processed;
the image segmentation module is used for carrying out image segmentation on an interested region in the target part image to obtain a mask image of the interested region;
and the target grading module is used for inputting the target part image and the mask image of the interested region into the twin network model to obtain a grading result of the target part image.
In a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement an image ranking method according to any of the embodiments of the invention.
In a fourth aspect, embodiments of the present invention further provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform the image grading method according to any of the embodiments of the present invention.
The method comprises the steps of extracting a target part image in an image to be processed by acquiring the image to be processed, and determining a target part in the image to be processed; further, carrying out image segmentation of an interested area in the target part image to obtain a mask image of the interested area; and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image. According to the technical scheme, when image classification is carried out, the feature information of the target part image and the mask image of the interested region can be effectively learned by means of the twin network model, so that the similarity of the target part image and the mask image of the interested region is measured, the grade of the target part image is accurately judged based on the similarity, automatic classification is realized, and the image processing efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, a brief description is given below of the drawings used in describing the embodiments. It should be clear that the described figures are only views of some of the embodiments of the invention to be described, not all, and that for a person skilled in the art, other figures can be derived from these figures without inventive effort.
Fig. 1 is a schematic flowchart of an image classification method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating an image classification method according to a second embodiment of the present invention;
fig. 3 is a schematic flowchart of an image classification method according to a third embodiment of the present invention;
FIG. 4 is a schematic structural diagram of an image classification apparatus according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention.
It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1 is a flowchart of an image classification method according to an embodiment of the present invention, where the embodiment is applicable to a case where an image is automatically classified through a neural network model, and the method may be executed by an image classification apparatus according to an embodiment of the present invention, where the apparatus may be implemented by software and/or hardware, and the apparatus may be configured on an electronic computing device, such as a desktop computer or a server. As shown in fig. 1, the image classification method of the present embodiment may specifically include the following steps:
s110, acquiring an image to be processed, and extracting a target part image in the image to be processed.
The image to be processed may be an image including the target portion. The type, content, and the like of the image to be processed are not particularly limited herein. Alternatively, the image to be processed includes a medical image or the like. Typically, the medical image may be a clinical medical image such as a skin image, a Computed Tomography (CT) image, a Magnetic Resonance (MR) image, a Positron Emission Tomography (PET) image, or the like. Illustratively, the image to be processed may be an image of an epidermal lesion, an image of an appendicitis lesion, or the like. Specifically, the image to be processed includes a target portion and a non-target portion, and a target portion image can be extracted from the image to be processed by a target extraction method.
In the present embodiment, the specific target portion is not limited, and the desired target portion may be identified according to the identification requirement of the target portion, and the target portion may be, for example, the back, the chest, or the like. In some embodiments, the target site includes a head. Accordingly, the image to be processed is an epidermal lesion image, and the epidermal lesion image includes a plurality of body parts, for example, body parts such as a head, a hand, or a leg. If the target region is a head, it is necessary to extract an image related to the head from the epidermal lesion image as a target region image.
In the embodiment of the invention, one, two or more images to be processed are acquired. Optionally, the acquiring the image to be processed includes: the method comprises the steps of acquiring an image to be processed containing a target part in real time based on an image acquisition device, or acquiring the image to be processed containing the target part from a preset storage position, or receiving the image to be processed containing the target part sent by the target device. The storage position of the image to be processed is not limited, the image to be processed can be set according to actual requirements, and the image to be processed can be directly acquired from the corresponding storage position when needed.
S120, carrying out image segmentation on the region of interest in the target part image to obtain a mask image of the region of interest.
Wherein the region of interest refers to selecting an image region from the target portion image, the region of interest being a focus of a user's analysis on the target portion image, and in some embodiments, the region of interest may be a lesion region. Masks are areas or processes that control image processing by masking an image, pattern or object, either completely or partially, with a selected image, pattern or object. The mask image refers to an image that is partially or completely occluded. In the image processing, the mask may be a two-dimensional matrix array or a multi-valued image, which is not limited in this embodiment.
Specifically, the region of interest in the target portion image is divided from the target image, so that the target portion image is divided into the region of interest and the region of non-interest, and the region of non-interest is blocked, so that the target portion image only contains the image content of the region of interest. Illustratively, the target part image is a head epidermis image containing a focus, the region of interest is a focus region, and the head epidermis image only contains the picture content of the focus region by shielding a normal skin region; illustratively, the target part image is a head epidermis image containing a lesion, the region of interest is a normal skin region, and the head epidermis image only contains picture content of the normal skin region by blocking the lesion region, wherein the specific type of the region of interest may be set according to requirements. In the embodiment, a smaller image area is defined by segmenting the image of the region of interest, so that the workload can be greatly reduced in the subsequent image processing process, non-relevant information is removed, and the image processing efficiency is improved.
S130, inputting the target part image and the mask image of the interested area into a twin network model to obtain a grading result of the target part image.
The twin network refers to a twin neural network (also known as a simense neural network), and is a coupling framework established based on two artificial neural networks. It should be noted that the image classification in the present embodiment refers to classification of a region of interest in an image, and for example, classification is performed according to the size of the region of interest in a target region image, the color shade, or the like.
Specifically, the target part image and the mask image of the region of interest are input into the twin network model trained in advance as input data, so that the classification result of the target part image corresponding to the target part image is obtained and is output from the twin network model as output data, and the image can be classified automatically with high efficiency and accuracy.
On the basis of the above embodiment, the twin network model includes a first feature extraction module, a second feature extraction module, and a hierarchical identification module, where the first feature extraction module is configured to perform feature extraction on the target region image, the second feature extraction module is configured to perform feature extraction on the mask image of the region of interest, the hierarchical identification module is configured to perform hierarchical identification according to the features extracted by the first feature extraction module and the second feature extraction module, and the hierarchical identification module includes at least two output nodes corresponding to hierarchical results of at least two levels.
The twin network may be composed of two identical neural networks, that is, the first feature extraction module and the second feature extraction module are two modules with the same function, and are used for performing feature extraction on the image, and only the input data is different. The classification recognition module can perform similarity calculation on the features extracted by the first feature extraction module and the second feature extraction module, and judge a classification result based on the numerical value of the similarity, so that the target part image can be classified more accurately.
In the embodiment of the present invention, the first feature extraction module may perform feature extraction on the input target region image. Specifically, the input target portion image is mapped to a new space, and the high-dimensional target portion image is converted into low-dimensional feature data, so that effective image features are extracted. The second feature extraction module may perform feature extraction on the input mask image of the region of interest. Specifically, the input mask image of the region of interest is mapped to a new space, and the mask image of the high-dimensional region of interest is converted into low-dimensional feature data, so that effective image features are extracted. The hierarchical identification module calculates the distance between the two features according to the features extracted by the first feature extraction module and the second feature extraction module, namely the similarity between the target part image and the mask image of the region of interest; and judging the grading result based on the numerical value of the similarity, wherein the numerical value range of the similarity is between 0 and 0.1, the corresponding grading result is a mild grade, the numerical value range of the similarity is between 0.8 and 1, and the corresponding grading result is a severe grade.
For example, the first feature extraction module and the second feature extraction module may be configured by a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or the like. The image is subjected to feature extraction through the convolutional neural network or the cyclic neural network, so that the image feature extraction method has good feature extraction capability, and when the image feature extraction method is applied to a twin network model, the feature extraction capability of the twin network model can be improved, and the grade can be accurately classified.
On the basis of the above embodiment, the twin network model further includes a self-attention module, and the self-attention module is configured to predict weights of the output features of the first feature extraction module and the second feature extraction module, and perform weighting processing on the output features of the first feature extraction module and the second feature extraction module.
Wherein, the self-attention module can further extract important features from the features output by the first feature extraction module and the second feature extraction module, and capture the internal correlation of the features in the target part image and the mask image of the region of interest. The self-Attention module includes, but is not limited to, SENET (Squeeze-and-Excitation Networks) algorithm, CBAM (volumetric Block Attention module) algorithm, and SKNet (selective Kernel Networks) algorithm. In some embodiments, the input end of the self-attention module is respectively connected with the first feature extraction module and the second feature extraction module, and the output end of the self-attention module is connected with the hierarchical identification module.
Specifically, the self-attention module predicts and acquires the weight of the output features of the first feature extraction module and the second feature extraction module in a learning mode so as to enhance useful features and suppress unimportant features. And the output characteristics of the first characteristic extraction module and the second characteristic extraction module are weighted to finish the recalibration of the output characteristics of the first characteristic extraction module and the second characteristic extraction module so as to improve the characteristic extraction capability of the twin network model. And in the case of limited computing power, the self-attention module can solve the information overload problem.
On the basis of the embodiment, a skeleton of the twin network model is built through a Backbone.
The Backbone network is a kind of Backbone network in the neural network. Illustratively, the backbone network adopts the resnext-50 as the backbone network, and the skeleton of the twin network model is built through the resnext-50, so that the accuracy of grading the twin network model can be improved on the premise of not increasing the complexity of parameters, and meanwhile, the number of hyper-parameters is reduced.
Illustratively, the target portion image is a human head image, the mask image of the region of interest is a normal skin segmentation image, and the normal skin segmentation image is input into a pre-trained twin network model as input data, wherein the twin network model comprises two CNN networks, an attention module SEnet and a main network next-50. Before the full connection layer of the twin network model, the characteristics of the two paths are weighted and fused, the weight corresponding to the human head picture is 0.5, the weight corresponding to the normal skin segmentation picture is 1, the output node of the full connection layer is set to be 5, and the output node corresponds to 5 levels of the focus degree. And if the historical experience shows that the picture of the normal skin segmentation is more important to the twin network model, the weight corresponding to the picture of the normal skin segmentation is greater than the weight corresponding to the picture of the human head. When the twin network model is trained, labeling is carried out according to the size grade of the focus area in the human head picture. The twin network model finally outputs a level of lesion area size on the human head, for example, level 1 (large area lesion occurrence), or level 5 (micro area lesion occurrence).
The method comprises the steps of extracting a target part image in an image to be processed by acquiring the image to be processed; carrying out image segmentation on an interested region in a target part image to obtain a mask image of the interested region; and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image. According to the technical scheme, when image classification is carried out, the characteristics of the target part image and the mask image of the interested region can be effectively learned by means of the twin network model, so that the similarity of the target part image and the mask image of the interested region is measured, the grade of the target part image is accurately judged based on the similarity, automatic classification is realized, and the image processing efficiency is improved.
Example two
Fig. 2 is a flowchart of an image classification method according to a second embodiment of the present invention. The technical scheme of the embodiment of the invention is further refined on the basis of the embodiment. Optionally, the extracting the target portion image in the image to be processed includes: detecting a target part of the image to be processed; and if the detection result of the image to be processed comprises the target part, extracting the target part image according to the parameter information of the target part of the detection result.
As shown in fig. 2, the method of the embodiment of the present invention specifically includes the following steps:
s210, acquiring an image to be processed, and detecting a target part of the image to be processed.
The task of the detection of the target portion is, among other things, to find all objects of interest in the image, determining their category and location. The method for detecting the target part of the image to be processed can include, but is not limited to, a traditional target detection method and a deep learning detection model method. The conventional target detection method generally comprises three stages: firstly, some candidate regions are selected on a given image, then the regions are subjected to feature extraction, and finally, a trained classifier is used for classification. The deep learning detection model can be specifically a Yolov4 algorithm or a Fast R-CNN algorithm.
On the basis of the above embodiment, the detecting of the target region of the image to be processed includes: and inputting the image to be processed to a part detection module to obtain a detection result output by the part detection module, wherein the detection result comprises classification information of a target part and parameter information of the target part.
The classification information is to classify a target part and a non-target part in the image to be processed, for example, the image to be processed includes a head and other body parts, if the detected target part is a head, the detected head is put into the target part for classification, and the other body parts are put into the non-target part for classification. The parameter information of the target site may include position information of the target site, a target site score. The position information of the target portion is used to extract the target portion from the image to be processed, and different target portions may correspond to different types of position information, which is not limited herein. Taking the target portion as a header, the parameter information of the target portion may include: a first location point left _ point, a second location point top _ point, a first size parameter width, and a second size parameter height. The target portion score is used to represent the confidence of the target portion, for example, if the target portion score is greater than a preset score threshold, it is determined that the currently detected target portion is valid, and the detection result of the image to be processed includes the target portion.
In the embodiment of the present invention, a part detection module is used for executing a method for detecting a target part, and specifically, based on a plurality of sets of training sample data, a deep learning model established in advance is trained to generate a part detection module having a classification function, where the training sample data includes a sample to-be-processed image and a sample target part image corresponding to the sample to-be-processed image; the image to be processed is input to the pre-trained part detection module, the classification information of the target part and the parameter information of the target part can be obtained, the automatic detection of the target part is realized, and the image processing efficiency is improved.
S220, if the detection result of the image to be processed includes the target part, extracting the target part image according to the parameter information of the target part of the detection result.
Specifically, when the detection result of the image to be processed includes the target portion, the position area of the target portion in the image to be processed can be obtained according to the position information of the target portion in the parameter information, and the target portion image is extracted according to the position area of the target portion in the image to be processed.
Illustratively, according to the position information of the target part, a crop instruction is used for removing a non-target part region in the image to be processed, the target part region is reserved, and the target part image is generated, wherein the crop instruction is used for clipping pictures, unnecessary information in the image is removed, and only a required part is reserved. According to the embodiment, the target part image can be automatically extracted through the crop instruction, and the image processing efficiency is improved.
And S230, carrying out image segmentation on the region of interest in the target part image to obtain a mask image of the region of interest.
In this embodiment, the region of interest in the target region image may be segmented based on an image segmenter. For example, the target region image is input to an image segmenter to obtain a mask image of the region of interest.
S240, inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image.
The embodiment of the invention provides an image grading method, which comprises the steps of detecting a target part of an image to be processed by acquiring the image to be processed, and extracting a target part image according to parameter information of the target part of a detection result if the detection result of the image to be processed comprises the target part; carrying out image segmentation of an interested region in the target part image to obtain a mask image of the interested region; and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image. According to the technical scheme, when image classification is carried out, the characteristics of the target part image and the mask image of the interested region can be effectively learned by means of the twin network model, so that the similarity between the target part image and the mask image of the interested region is measured, the grade of the target part image is accurately judged based on the similarity, automatic classification is realized, and the image processing efficiency is improved.
EXAMPLE III
Fig. 3 is a flowchart of an image classification method according to a third embodiment of the present invention. The technical scheme of the embodiment of the invention is further refined on the basis of the embodiment. Optionally, the image segmentation of the region of interest in the target portion image to obtain the mask image of the region of interest includes: detecting a region of interest of the target part image; and if the target part image comprises the region of interest, performing image segmentation on the region of interest of the target part image to obtain a mask image of the region of interest.
As shown in fig. 3, the method of the embodiment of the present invention specifically includes the following steps:
s310, acquiring an image to be processed, and extracting a target part image in the image to be processed.
And S320, detecting the region of interest of the target part image.
The method for detecting the region of interest of the target region image may include a classification model method. The classification model may specifically be an AlexNet algorithm or a ResNet residual network algorithm.
On the basis of the foregoing embodiment, the detecting of the region of interest of the target region image includes: inputting the target part image into a classification model of the interested region to obtain the identification probability of the interested region; and if the identification probability is greater than a preset threshold value, determining that the target part image comprises an interested region.
The classification model is a deep learning model trained in advance, and specifically, the deep learning classification model established in advance is trained based on a plurality of groups of training sample data to generate the classification model, wherein the training sample data includes a sample target portion image and a sample region-of-interest image corresponding to the target portion image. For example, the classification model is subjected to two classification training with a focus human head and a normal human head.
Illustratively, the region of interest may be a human head lesion. Inputting a target part image of a target part, namely a human head, into a pre-trained deep learning classification model, obtaining the recognition probability of the human head focus through the deep learning classification model, and if the recognition probability is greater than a preset threshold value of 0.5, determining that the human head includes the human head focus, namely that the human head has the focus. According to the method and the device, the automatic detection of the region of interest is realized through the deep learning classification model, and the image processing efficiency is improved.
S330, if the target part image comprises the region of interest, performing image segmentation on the region of interest of the target part image to obtain a mask image of the region of interest.
S340, inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image.
The embodiment of the invention provides an image grading method, which comprises the steps of extracting a target part image in an image to be processed by acquiring the image to be processed; detecting an interested region of the target part image, and if the target part image comprises the interested region, performing image segmentation on the interested region of the target part image to obtain a mask image of the interested region; and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image. According to the technical scheme, when image classification is carried out, the characteristics of the target part image and the mask image of the interested region can be effectively learned by means of the twin network model, so that the similarity between the target part image and the mask image of the interested region is measured, the grade of the target part image is accurately judged based on the similarity, automatic classification is realized, and the image processing efficiency is improved.
Example four
Fig. 4 is a schematic structural diagram of an image classification apparatus according to a fourth embodiment of the present invention, where the image classification apparatus according to the fourth embodiment of the present invention may be implemented by software and/or hardware, and may be configured in a terminal and/or a server to implement the image classification method according to the fourth embodiment of the present invention. The device may specifically comprise: an object extraction module 410, an image segmentation module 420, and an object ranking module 430.
The target extraction module 410 is configured to obtain an image to be processed, and extract a target portion image in the image to be processed; an image segmentation module 420, configured to perform image segmentation on a region of interest in the target portion image to obtain a mask image of the region of interest; and the target grading module 430 is configured to input the target portion image and the mask image of the region of interest into a twin network model, so as to obtain a grading result of the target portion image.
The embodiment of the invention provides an image grading device, which extracts a target part image in an image to be processed by acquiring the image to be processed; carrying out image segmentation of an interested region in the target part image to obtain a mask image of the interested region; and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image. According to the technical scheme, when image classification is carried out, the characteristics of the target part image and the mask image of the interested region can be effectively learned by means of the twin network model, so that the similarity between the target part image and the mask image of the interested region is measured, the grade of the target part image is accurately judged based on the similarity, automatic classification is realized, and the image processing efficiency is improved.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the target extraction module 410 may include:
the image detection unit is used for detecting a target part of the image to be processed;
and the target part image extracting unit is used for extracting the target part image according to the parameter information of the target part of the detection result if the detection result of the image to be processed comprises the target part.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the target image detection unit may be configured to:
and inputting the image to be processed to a part detection module to obtain a detection result output by the part detection module, wherein the detection result comprises classification information of a target part and parameter information of the target part.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the image segmentation module 420 may include:
the interested region detection unit is used for detecting the interested region of the target part image;
and the mask image acquisition unit is used for carrying out image segmentation on the region of interest of the target part image to obtain a mask image of the region of interest if the target part image comprises the region of interest.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the region of interest detection unit may be specifically configured to:
inputting the target part image into a classification model of the interested region to obtain the identification probability of the interested region;
and if the identification probability is greater than a preset threshold value, determining that the target part image comprises an interested region.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the twin network model includes a first feature extraction module, a second feature extraction module, and a hierarchical identification module, where the first feature extraction module is configured to perform feature extraction on the target region image, the second feature extraction module is configured to perform feature extraction on the mask image of the region of interest, the hierarchical identification module is configured to perform hierarchical identification according to the features extracted by the first feature extraction module and the second feature extraction module, and the hierarchical identification module includes at least two output nodes corresponding to hierarchical results of at least two levels.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the twin network model further includes a self-attention module, where the self-attention module is configured to predict weights of output features of the first feature extraction module and the second feature extraction module, and perform weighting processing on the output features of the first feature extraction module and the second feature extraction module.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the target site includes a head.
The image grading device provided by the embodiment of the invention can execute the image grading method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
Fig. 5 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention. The electronic device 12 shown in fig. 5 is only an example and should not bring any limitation to the function and the scope of use of the embodiment of the present invention.
As shown in FIG. 5, electronic device 12 is embodied in the form of a general purpose computing device. The components of electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The electronic device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, implementing an image classification method provided by the present embodiment.
EXAMPLE six
An embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a method of image ranking, the method comprising:
acquiring an image to be processed, and extracting a target part image in the image to be processed;
carrying out image segmentation of an interested region in the target part image to obtain a mask image of the interested region;
and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (11)
1. An image ranking method, comprising:
acquiring an image to be processed, and extracting a target part image in the image to be processed;
carrying out image segmentation of an interested region in the target part image to obtain a mask image of the interested region;
and inputting the target part image and the mask image of the region of interest into a twin network model to obtain a grading result of the target part image.
2. The method according to claim 1, wherein the extracting the target portion image in the image to be processed comprises:
detecting a target part of the image to be processed;
and if the detection result of the image to be processed comprises the target part, extracting the target part image according to the parameter information of the target part of the detection result.
3. The method according to claim 2, wherein the detecting the target portion of the image to be processed comprises:
and inputting the image to be processed to a part detection module to obtain a detection result output by the part detection module, wherein the detection result comprises classification information of a target part and parameter information of the target part.
4. The method according to claim 1, wherein the image segmentation of the region of interest in the target portion image to obtain the mask image of the region of interest comprises:
detecting a region of interest of the target part image;
and if the target part image comprises the region of interest, performing image segmentation on the region of interest of the target part image to obtain a mask image of the region of interest.
5. The method according to claim 4, wherein the detecting of the region of interest of the target region image comprises:
inputting the target part image into a classification model of the interested region to obtain the identification probability of the interested region;
and if the identification probability is greater than a preset threshold value, determining that the target part image comprises an interested region.
6. The method according to claim 1, wherein the twin network model comprises a first feature extraction module, a second feature extraction module and a hierarchical identification module, wherein the first feature extraction module is used for extracting features of the target region image, the second feature extraction module is used for extracting features of the mask image of the region of interest, the hierarchical identification module is used for performing hierarchical identification according to the features extracted by the first feature extraction module and the second feature extraction module, and the hierarchical identification module comprises at least two output nodes corresponding to at least two levels of hierarchical results.
7. The method of claim 6, further comprising a self-attention module for predicting weights of output features of the first and second feature extraction modules, and performing weighting processing on the output features of the first and second feature extraction modules.
8. The method of claim 1, wherein the target site comprises a head.
9. An image grading apparatus, comprising:
the target extraction module is used for acquiring an image to be processed and extracting a target part image in the image to be processed;
the image segmentation module is used for carrying out image segmentation on an interested region in the target part image to obtain a mask image of the interested region;
and the target grading module is used for inputting the target part image and the mask image of the interested region into the twin network model to obtain a grading result of the target part image.
10. An apparatus, characterized in that the apparatus comprises:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the image ranking method of any of claims 1-8.
11. A storage medium containing computer executable instructions for performing the image grading method of any of claims 1-8 when executed by a computer processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110641027.9A CN113222989A (en) | 2021-06-09 | 2021-06-09 | Image grading method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110641027.9A CN113222989A (en) | 2021-06-09 | 2021-06-09 | Image grading method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113222989A true CN113222989A (en) | 2021-08-06 |
Family
ID=77083290
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110641027.9A Pending CN113222989A (en) | 2021-06-09 | 2021-06-09 | Image grading method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113222989A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116912247A (en) * | 2023-09-13 | 2023-10-20 | 威海市博华医疗设备有限公司 | Medical image processing method and device, storage medium and electronic equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348477A (en) * | 2019-06-04 | 2019-10-18 | 上海联影智能医疗科技有限公司 | Medical image processing method, storage medium and computer equipment |
| CN110569725A (en) * | 2019-08-05 | 2019-12-13 | 华中科技大学 | Gait recognition system and method for deep learning based on self-attention mechanism |
| CN111160120A (en) * | 2019-12-11 | 2020-05-15 | 重庆邮电大学 | Fast R-CNN article detection method based on transfer learning |
| CN111275703A (en) * | 2020-02-27 | 2020-06-12 | 腾讯科技（深圳）有限公司 | Image detection method, image detection device, computer equipment and storage medium |
| CN111680632A (en) * | 2020-06-10 | 2020-09-18 | 深延科技(北京)有限公司 | Smoke and fire detection method and system based on deep learning convolutional neural network |
| CN111899265A (en) * | 2020-06-24 | 2020-11-06 | 上海联影智能医疗科技有限公司 | Image analysis method, image analysis device, computer equipment and storage medium |
| CN112001902A (en) * | 2020-08-19 | 2020-11-27 | 上海商汤智能科技有限公司 | Defect detection method and related device, equipment and storage medium |
| CN113762305A (en) * | 2020-11-27 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for determining alopecia type |
-
2021
- 2021-06-09 CN CN202110641027.9A patent/CN113222989A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348477A (en) * | 2019-06-04 | 2019-10-18 | 上海联影智能医疗科技有限公司 | Medical image processing method, storage medium and computer equipment |
| CN110569725A (en) * | 2019-08-05 | 2019-12-13 | 华中科技大学 | Gait recognition system and method for deep learning based on self-attention mechanism |
| CN111160120A (en) * | 2019-12-11 | 2020-05-15 | 重庆邮电大学 | Fast R-CNN article detection method based on transfer learning |
| CN111275703A (en) * | 2020-02-27 | 2020-06-12 | 腾讯科技（深圳）有限公司 | Image detection method, image detection device, computer equipment and storage medium |
| CN111680632A (en) * | 2020-06-10 | 2020-09-18 | 深延科技(北京)有限公司 | Smoke and fire detection method and system based on deep learning convolutional neural network |
| CN111899265A (en) * | 2020-06-24 | 2020-11-06 | 上海联影智能医疗科技有限公司 | Image analysis method, image analysis device, computer equipment and storage medium |
| CN112001902A (en) * | 2020-08-19 | 2020-11-27 | 上海商汤智能科技有限公司 | Defect detection method and related device, equipment and storage medium |
| CN113762305A (en) * | 2020-11-27 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for determining alopecia type |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116912247A (en) * | 2023-09-13 | 2023-10-20 | 威海市博华医疗设备有限公司 | Medical image processing method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11861829B2 (en) | Deep learning based medical image detection method and related device | |
| WO2021017372A1 (en) | Medical image segmentation method and system based on generative adversarial network, and electronic equipment | |
| CN110490850B (en) | Lump region detection method and device and medical image processing equipment | |
| CN110689038A (en) | Training method and device of neural network model and medical image processing system | |
| CN110796199B (en) | Image processing method and device and electronic medical equipment | |
| CN109389129A (en) | A kind of image processing method, electronic equipment and storage medium | |
| CN112102237A (en) | Brain tumor recognition model training method and device based on semi-supervised learning | |
| CN109754007A (en) | Peplos intelligent measurement and method for early warning and system in operation on prostate | |
| US9330336B2 (en) | Systems, methods, and media for on-line boosting of a classifier | |
| CN111754453A (en) | Pulmonary tuberculosis detection method and system based on chest radiography image and storage medium | |
| Fan et al. | GFNet: automatic segmentation of COVID-19 lung infection regions using CT images based on boundary features | |
| CN114549557A (en) | Portrait segmentation network training method, device, equipment and medium | |
| CN116188392A (en) | Image processing method, computer-readable storage medium, and computer terminal | |
| CN116802683A (en) | Image processing method and system | |
| Guo et al. | LLTO: towards efficient lesion localization based on template occlusion strategy in intelligent diagnosis | |
| CN113222989A (en) | Image grading method and device, storage medium and electronic equipment | |
| CN111127432B (en) | Medical image detection method, device, equipment and storage medium | |
| CN113096080A (en) | Image analysis method and system | |
| Li et al. | Fingertip blood collection point localization research based on infrared finger vein image segmentation | |
| CN112750124B (en) | Model generation method, image segmentation method, model generation device, image segmentation device, electronic equipment and storage medium | |
| CN112530554B (en) | Scanning positioning method and device, storage medium and electronic equipment | |
| CN111598144B (en) | Training method and device for image recognition model | |
| US11972571B2 (en) | Method for image segmentation, method for training image segmentation model | |
| CN113408596B (en) | Pathological image processing method and device, electronic equipment and readable storage medium | |
| US20220036561A1 (en) | Method for image segmentation, method for training image segmentation model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |