CN111340820B - Image segmentation method and device, electronic equipment and storage medium - Google Patents
Image segmentation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111340820B CN111340820B CN202010085655.9A CN202010085655A CN111340820B CN 111340820 B CN111340820 B CN 111340820B CN 202010085655 A CN202010085655 A CN 202010085655A CN 111340820 B CN111340820 B CN 111340820B
- Authority
- CN
- China
- Prior art keywords
- image
- feature
- segmented
- level
- compact
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000003709 image segmentation Methods 0.000 title claims abstract description 35
- 230000011218 segmentation Effects 0.000 claims abstract description 38
- 230000008569 process Effects 0.000 claims description 19
- 230000001965 increasing effect Effects 0.000 claims description 14
- 230000004927 fusion Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 5
- 238000002372 labelling Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000005457 optimization Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 210000005252 bulbus oculi Anatomy 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000002059 diagnostic imaging Methods 0.000 description 2
- 230000004069 differentiation Effects 0.000 description 2
- 210000004072 lung Anatomy 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 210000001508 eye Anatomy 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
Abstract
An image segmentation method is applied to the technical field of medical image processing and comprises the following steps: the method comprises the steps of coding an image to be segmented step by step to obtain a highest-level feature image of the image to be segmented, inputting the highest-level feature image to a compact feature learning model to obtain a compact feature image, decoding the compact feature image in combination with the highest-level feature image step by step to restore spatial information of the compact feature image to obtain a target image to be segmented, predicting the class to be segmented of each pixel in the target image to be segmented to obtain a segmentation prediction image of the target image to be segmented, and extracting a prediction boundary of the target image to be segmented according to the segmentation prediction image. The application also discloses an image segmentation device, electronic equipment and a storage medium, which can improve the image segmentation accuracy and reduce the noise influence in image segmentation.
Description
Technical Field
The present application relates to the field of medical image processing technologies, and in particular, to an image segmentation method and apparatus, an electronic device, and a storage medium.
Background
Limited by medical imaging methods and medical imaging equipment, medical images often have the characteristic of edge blurring, which causes difficulty in labeling work of doctors, and particularly, labeling of boundaries depends heavily on experience and competent standards of doctors. The labeling differentiation exists between multiple labeling of the same doctor and between different labeling of doctors, so that the inherent noise problem caused by the labeling differentiation is often ignored in the automatic segmentation model training process.
In the existing research method, the boundary optimization mode is usually to give higher weight to boundary pixels in the training process, which enhances the sensitivity of the model to the boundary to a certain extent, but also increases the influence caused by noise in the training process.
Disclosure of Invention
The present application provides an image segmentation method, an image segmentation apparatus, an electronic device, and a storage medium, which can improve image segmentation accuracy and reduce noise influence in image segmentation.
In order to achieve the above object, a first aspect of embodiments of the present application provides an image segmentation method, including:
inputting an image to be segmented into a coding module of a semantic segmentation network for coding step by step, sequentially reducing spatial information and increasing semantic information of the multilayer characteristic image, and acquiring a characteristic image with minimum spatial information and maximum semantic information in the multilayer characteristic image as a highest-level characteristic image;
sending the highest-level feature image to a compact feature module, calculating feature centers of various categories to be segmented through rough segmentation prediction results in the compact feature learning module, and then performing feature reconstruction on the highest-level feature image according to the similarity of each pixel in the highest-level feature image and each feature center to obtain a compact feature image;
combining the obtained compact feature map with the highest-level feature map to obtain a first-level compact feature image, obtaining a feature image in the encoding process with the same spatial information as the first-level compact feature image, performing feature fusion on the first-level compact feature image and the feature image to obtain a second-level compact feature image, and accordingly obtaining an Nth-level compact feature image, wherein the spatial information of the Nth-level compact feature image is the same as the spatial information of the image to be segmented, and taking the Nth-level compact feature image as a target image to be segmented;
predicting the class to be segmented of each pixel in the target image to be segmented to obtain a segmentation prediction image of the target image to be segmented;
extracting a prediction boundary of the target image to be segmented according to the segmentation prediction graph;
optionally, the inputting the highest-level feature image into a compact feature learning model to obtain a compact feature image includes:
predicting the feature center of each category to be segmented according to the highest-level feature image;
according to a self-adaptive attention mechanism, calculating the similarity between the feature of each pixel in the highest-level feature image and each feature center;
and reconstructing the highest-level feature image according to the similarity between the feature of each pixel in the highest-level feature image and each feature center to obtain the compact feature image.
Optionally, the extracting, according to the segmentation prediction map, the prediction boundary of the target image to be segmented includes:
and extracting boundary pixels from the segmentation prediction graph of the target image to be segmented, wherein the types of the classes to be segmented of all the pixels in the preset neighborhood of the boundary pixels comprise at least two types.
Respectively acquiring the number of boundary pixels in each pixel preset neighborhood;
taking the number of boundary pixels in each pixel preset neighborhood as the weight of the corresponding pixel;
calculating the loss of a preset neighborhood of each pixel according to the weight of each pixel;
and obtaining the predicted boundary of the target image to be segmented according to the loss of the preset neighborhood of each pixel.
Optionally, the step-by-step decoding the compact feature image in combination with the highest-level feature image to restore the spatial information of the compact feature image, and obtaining the target image to be segmented includes:
performing feature fusion on the compact feature image and the highest-level feature image to obtain a first-level compact feature image;
acquiring a characteristic image in the encoding process which is the same as the first-stage compact characteristic image space information;
performing feature fusion on the first-level compact feature image and the feature image to obtain a second-level compact feature image, and accordingly obtaining an Nth-level compact feature image, wherein the spatial information of the Nth-level compact feature image is the same as that of the image to be segmented;
and obtaining a target image to be segmented by the Nth-level compact characteristic image through a prediction layer.
Optionally, the spatial information of the first to N-level compact feature images is sequentially increased, and the semantic information is sequentially increased.
Optionally, when the number of boundary pixels in the preset neighborhood of the pixel is 0, the weight of the pixel is assigned to be 1.
Optionally, the step-by-step encoding the image to be segmented to obtain the highest-level feature image of the image to be segmented includes:
the method comprises the steps of coding an image to be segmented step by step to obtain a multilayer characteristic image, wherein spatial information of the multilayer characteristic image is sequentially reduced, and semantic information is sequentially increased;
and acquiring the characteristic image with the minimum spatial information and the maximum semantic information in the multilayer characteristic images as the highest-level characteristic image of the image to be segmented.
A second aspect of the embodiments of the present application provides an image segmentation apparatus, including:
the encoding module is used for inputting an image to be segmented into an encoding module of a semantic segmentation network for encoding step by step, the spatial information of the multilayer characteristic images is sequentially reduced, the semantic information is sequentially increased, and the characteristic image with the minimum spatial information and the maximum semantic information in the multilayer characteristic images is obtained as the highest-level characteristic image;
the input module is used for sending the highest-level feature image to the compact feature module, calculating feature centers of various types to be segmented through rough segmentation prediction results in the compact feature learning module, and then performing feature reconstruction on the highest-level feature image according to the similarity between each pixel in the highest-level feature image and each feature center to obtain a compact feature image;
the decoding module is used for combining the obtained compact characteristic image with the highest-level characteristic image to obtain a first-level compact characteristic image, obtaining a characteristic image in the encoding process, the spatial information of which is the same as that of the first-level compact characteristic image, performing characteristic fusion on the first-level compact characteristic image and the characteristic image to obtain a second-level compact characteristic image, and accordingly obtaining an Nth-level compact characteristic image, wherein the spatial information of the Nth-level compact characteristic image is the same as that of the image to be segmented, and taking the Nth-level compact characteristic image as a target image to be segmented;
and the prediction module is used for predicting the class to be segmented of each pixel in the target image to be segmented to obtain a segmentation prediction image of the target image to be segmented.
A third aspect of embodiments of the present application provides an electronic device, including:
the image segmentation method is characterized in that the processor executes the program to implement the image segmentation method provided by the first aspect of the embodiment of the present application.
A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the image segmentation method provided in the first aspect of the embodiments of the present application.
As can be seen from the foregoing embodiments of the present application, the image segmentation method, apparatus, electronic device, and storage medium provided in the present application encode an image to be segmented step by step to obtain a top-level feature image of the image to be segmented, input the top-level feature image to a compact feature learning model to obtain a compact feature image, decode the compact feature image in combination with the top-level feature image step by step to restore spatial information of the compact feature image to obtain a target image to be segmented, predict categories to be segmented of pixels in the target image to be segmented to obtain a segmentation prediction map of the target image to be segmented, extract a prediction boundary of the target image to be segmented according to the segmentation prediction map, thereby improving image segmentation accuracy and reducing noise influence in image segmentation.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flowchart of an image segmentation method according to an embodiment of the present application;
fig. 2 is a schematic diagram of a framework of an image segmentation method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a compact feature learning module provided by an embodiment of the present application;
fig. 4 is a schematic structural diagram of an image segmentation apparatus according to an embodiment of the present application;
fig. 5 shows a hardware structure diagram of an electronic device.
Detailed Description
In order to make the purpose, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic flowchart of an image segmentation method according to an embodiment of the present application, and fig. 2 is a schematic diagram of a framework of the image segmentation method according to an embodiment of the present application, where the method is applicable to an electronic device, where the electronic device includes: the method mainly comprises the following steps that electronic devices capable of performing data processing in moving, such as mobile phones, tablet computers, portable computers, intelligent watches, intelligent glasses and the like, and electronic devices capable of performing data processing in moving, such as desktop computers, all-in-one machines, intelligent televisions and the like, are adopted:
s101, coding an image to be segmented step by step to obtain a highest-level feature image of the image to be segmented;
s102, inputting the highest-level feature image to a compact feature learning model to obtain a compact feature image;
s103, combining the compact characteristic image with the highest-level characteristic image to decode step by step so as to restore the spatial information of the compact characteristic image and obtain a target image to be segmented;
s104, predicting the to-be-segmented category of each pixel in the to-be-segmented target image to obtain a segmentation prediction image of the to-be-segmented target image;
and S105, extracting the prediction boundary of the target image to be segmented according to the segmentation prediction graph.
In one embodiment of the present application, the step S101 includes:
the method comprises the steps of coding an image to be segmented step by step to obtain a multilayer characteristic image, wherein spatial information of the multilayer characteristic image is sequentially reduced, and semantic information is sequentially increased; and acquiring a characteristic image with minimum spatial information and maximum semantic information in the multilayer characteristic images as the highest-level characteristic image of the image to be segmented.
The spatial information of the image can be understood as the resolution of the image, and the semantic information can be understood as the feature information of the image.
In one embodiment of the present application, step S102 includes:
predicting the feature center of each category to be segmented according to the highest-level feature image; calculating the similarity between the feature of each pixel in the highest-level feature image and each feature center according to a self-adaptive attention mechanism; and reconstructing the highest-level feature image according to the similarity between the feature of each pixel in the highest-level feature image and each feature center to obtain the compact feature image. The obtained compact feature image enables pixels of the same category to be more compactly expressed in feature space, and pixels of different categories are more obviously different in feature space.
Specifically, a feature of the medical image is that there are few segmentation classes, often less than 5 classes, and the feature dimension learned by the network is very high, few hundreds of dimensions, and many thousands of dimensions, such feature expression is very redundant, and at the same time, it is very easy to over-fit to the inherent noise, in order to make the segmentation prediction obtained by the decisionThe segmentation boundary in the graph is more robust to noise, see FIG. 3, for high-dimensional features X ∈ RC*H*W(wherein C is the dimension of the feature, and H and W respectively represent the height and width of the feature), firstly, the relay supervision thought is utilized to predict, and a rough segmentation probability map P is obtainedinter∈RK*H*W(K represents the number of classes to be segmented), and then calculating the feature centers of the classes according to the rough segmentation probability graph:
wherein x isiThe feature at the i-position is represented,
f, which is calculated by representing the probability of the rough segmentation probability map at the i position belonging to the class c to be segmentedcRepresenting the feature center of class c.
After the feature centers of all the categories to be segmented are obtained, the features of the highest-level feature image are reconstructed according to the distance (similarity) from the features to the feature centers of all the categories to be segmented, and a compact feature learning model is enabled to learn a mapping mode of feature reconstruction by itself in combination with an adaptive attention mechanism, wherein the formula is as follows:
the above formula represents the adaptively learned feature xiAnd c feature center f of class to be segmentedcWhere θ, φ represents a convolution operation with a convolution kernel size of 1 × 1, the compact feature image x obtained after reconstructioncompactIs represented as follows:
xcompact＝∑x′ic·γ(fc)
in the above formula, gamma is also the convolution operation with convolution kernel size of 1 x 1, and the obtained new feature expression xcompactThe distance between the feature of each pixel in the image including the highest-level feature and the center of each feature is iteratedAfter learning, the class to be segmented which is close to the self is closer and farther, and the original class to be segmented which is farther, so that the effects that the feature expression in the same class to be segmented is more compact and the difference between different classes to be segmented is larger are achieved, and the robustness of the model to noise is further enhanced.
The category to be segmented refers to the type of the image in the picture to be segmented. For example, if the picture to be segmented is a chest picture, the image in the picture to be segmented includes a background, a heart and a lung, and the category to be segmented is the background, the heart and the lung; if the picture to be segmented is an eye piece, the image in the picture to be segmented comprises a background and an eyeball, and the class to be segmented is the background and the eyeball.
In one embodiment of the present application, the step S103 includes:
performing feature fusion on the compact feature image and the highest-level feature image to obtain a first-level compact feature image; acquiring a characteristic image in the encoding process which is the same as the first-stage compact characteristic image space information; performing feature fusion on the first-level compact feature image and the feature image to obtain a second-level compact feature image, and accordingly obtaining an Nth-level compact feature image, wherein the spatial information of the Nth-level compact feature image is the same as that of the image to be segmented; and taking the Nth-level compact characteristic image as a target image to be segmented.
In one embodiment of the application, the spatial information of the first-level to N-level compact feature images is sequentially increased, and the semantic information is sequentially increased.
In one embodiment of the present application, step S105 includes:
and extracting boundary pixels from the segmentation prediction graph of the target image to be segmented, wherein the types of the classes to be segmented of all the pixels in the preset neighborhood of the boundary pixels comprise at least two types. Respectively acquiring the number of boundary pixels in each pixel preset neighborhood; taking the number of boundary pixels in each pixel preset neighborhood as the weight of the corresponding pixel; calculating the loss of a preset neighborhood of each pixel according to the weight of each pixel; and obtaining the predicted boundary of the target image to be segmented according to the loss of the preset neighborhood of each pixel.
Specifically, the model training process is always affected by noise due to the labeling noise problem caused by boundary blurring, which is more obvious under the loss of cross entropy (cross entropy) which is a traditional optimization objective function, because the cross entropy function is a convex function and the gradient is non-boundary. The proportion of potentially noisy pixels in the loss is highlighted, thus disturbing the correct direction of update. The method is based on the priori knowledge that noise often exists at the boundary, the pixels with large information amount are given higher weight to the current model in the optimization process, the thought of course learning is adopted, the information amount of the samples which are too simple or difficult is not large for improving the current model capability, and the samples near the decision boundary are important to the model in the current iteration process. Based on this, the method proposes the concept of "pseudo boundary" (pseudo) and "boundary pixel" (boundary) to extract the region of interest in each step of the update process, and the specific operations are as follows:
firstly, in the t-th iteration process, obtaining a segmentation prediction graph P according to network output;
then, it is determined which pixels belong to the boundary pixels from the generated prediction map. The judgment rule is that if the prediction categories are inconsistent (i.e. more than two prediction categories are contained) in the K-neighborhood of the pixel, the pixel is a "boundary pixel". The formulation is expressed as follows:
where i denotes the ith pixel, Nb (i) denotes the pixel in the k neighborhood of the i pixel, piRepresenting a prediction class of an ith pixel
Is a binary determinant when p isiAnd pjThe same is 1, and the opposite is 0.
Then, the number of boundary pixels in the K neighborhood of the pixel i is counted as the weight of the pixel i, and in order to avoid information loss, the weight of 1 is given to pixels without boundary pixels in the K neighborhood.
In the above formula Wt(i) Represents the weight of pixel i during t iterations, Nb (i) represents the pixels in k neighborhood of i pixels, Ft(i) Indicating whether a pixel j within the k neighborhood of pixel i is a boundary pixel.
Then, the loss in the t iteration rounds is calculated, which is called domain loss:
wherein N is the number of pixels in the image, Wt(i) Represents the weight of pixel i, y represents the label class of pixel i,
representing the probability that pixel i is predicted as labeling class y.
Through the optimization strategy, the model updating process conforms to the optimization process of course learning, each step of model updating focuses more on the boundary of the current model judgment capability, so that the influence of boundary noise in the early updating process is reduced, and the boundary noise is gradually fitted to the actual boundary, so that the purposes of accelerating the convergence speed and enhancing the noise robustness of the model are achieved.
Referring to fig. 4, fig. 4 is a schematic structural diagram of an image segmentation apparatus according to an embodiment of the present application, which may be embedded in an electronic device, and the apparatus mainly includes:
the encoding module 201 is configured to encode an image to be segmented step by step to obtain a highest-level feature image of the image to be segmented;
an input module 202, configured to input the highest-level feature image to a compact feature learning model to obtain a compact feature image;
the decoding module 203 is configured to decode the compact feature image in combination with the highest-level feature image step by step to restore spatial information of the compact feature image, so as to obtain a target image to be segmented;
the prediction module 204 is configured to predict a to-be-segmented category of each pixel in the to-be-segmented target image, so as to obtain a segmentation prediction map of the to-be-segmented target image;
and the extracting module 205 is configured to extract the prediction boundary of the target image to be segmented according to the segmentation prediction map.
In one embodiment of the present application, the encoding module 201 includes:
the coding sub-module is used for coding the image to be segmented step by step to obtain a multilayer characteristic image, the spatial information of the multilayer characteristic image is sequentially reduced, and the semantic information is sequentially increased;
and the first acquisition submodule is used for acquiring the characteristic image with the minimum spatial information and the maximum semantic information in the multilayer characteristic image as the highest-level characteristic image of the image to be segmented.
In one embodiment of the present application, the input module 202 includes:
the prediction submodule is used for predicting the feature center of each category to be segmented according to the highest-level feature image;
the calculation submodule calculates the similarity between the feature of each pixel in the highest-level feature image and each feature center according to a self-adaptive attention mechanism;
and the reconstruction submodule is used for reconstructing the highest-level feature image according to the similarity between the feature of each pixel in the highest-level feature image and each feature center to obtain the compact feature image.
In one embodiment of the present application, the decoding module 203 comprises:
the fusion submodule is used for carrying out feature fusion on the compact feature image and the highest-level feature image to obtain a first-level compact feature image;
the second acquisition submodule is used for acquiring the characteristic image in the coding process, which is the same as the space information of the first-stage compact characteristic image;
the fusion sub-module is further configured to perform feature fusion on the first-level compact feature image and the feature image to obtain a second-level compact feature image, and accordingly obtain an nth-level compact feature image, where spatial information of the nth-level compact feature image is the same as spatial information of the image to be segmented;
and the determining submodule is used for taking the Nth-level compact characteristic image as a target image to be segmented.
In one embodiment of the application, the spatial information of the first-level to N-level compact feature images is sequentially increased, and the semantic information is sequentially increased.
In one embodiment of the present application, the extraction module 205 comprises:
and the extraction submodule is used for extracting boundary pixels from the segmentation prediction image of the target image to be segmented, and the types of the classes to be segmented of all the pixels in the preset neighborhood of the boundary pixels comprise at least two types.
The acquisition module is used for respectively acquiring the number of boundary pixels in each pixel preset neighborhood;
the weighting module is used for taking the number of boundary pixels in each pixel preset neighborhood as the weight of the corresponding pixel;
the calculation module is used for calculating the preset neighborhood loss of each pixel in the current iteration process according to the weight of each pixel;
and the determining module is used for obtaining the prediction boundary of the target image to be segmented according to the loss of the preset neighborhood of each pixel.
Referring to fig. 5, fig. 5 is a hardware structure diagram of an electronic device.
The electronic device described in this embodiment includes:
a memory 31, a processor 32 and a computer program stored on the memory 31 and executable on the processor, the processor implementing the image segmentation method described in the foregoing embodiments shown in fig. 1 to 2 when executing the program.
Further, the electronic device further includes:
at least one input device 33; at least one output device 34.
The memory 31, processor 32 input device 33 and output device 34 are connected by a bus 35.
The input device 33 may be a camera, a touch panel, a physical button, or a mouse. The output device 34 may specifically be a display screen.
The Memory 31 may be a high-speed Random Access Memory (RAM) Memory or a non-volatile Memory (non-volatile Memory), such as a disk Memory. The memory 31 is used for storing a set of executable program code, and the processor 32 is coupled to the memory 31.
Further, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium may be provided in the terminal in the foregoing embodiments, and the computer-readable storage medium may be the memory in the foregoing embodiment shown in fig. 5. The computer-readable storage medium has stored thereon a computer program which, when executed by a processor, implements the image segmentation method described in the foregoing embodiments shown in fig. 1 to 2. Further, the computer-readable storage medium may be various media that can store program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It should be noted that each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.
The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be substantially or partially embodied in the form of a software product, or all or part of the technical solution that contributes to the prior art.
It should be noted that, for the sake of simplicity, the above-mentioned method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present invention is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no acts or modules are necessarily required of the invention.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In view of the above description of the image segmentation method, apparatus, electronic device and storage medium provided by the present invention, those skilled in the art will recognize that there may be variations in the embodiments and applications of the concept of the present invention, and therefore the present description should not be construed as limiting the present invention.
Claims (7)
1. An image segmentation method, comprising:
inputting an image to be segmented into a coding module of a semantic segmentation network for coding step by step to obtain a multilayer characteristic image, wherein the spatial information of the multilayer characteristic image is reduced in sequence, the semantic information is increased in sequence, and the characteristic image with the minimum spatial information and the maximum semantic information in the multilayer characteristic image is obtained as a highest-level characteristic image;
sending the highest-level feature image to a compact feature module, calculating feature centers of various categories to be segmented through rough segmentation prediction results in the compact feature learning module, and then performing feature reconstruction on the highest-level feature image according to the similarity of each pixel in the highest-level feature image and each feature center to obtain a compact feature image;
combining the obtained compact feature map with the highest-level feature map to obtain a first-level compact feature image, obtaining a feature image in the encoding process with the same spatial information as the first-level compact feature image, performing feature fusion on the first-level compact feature image and the feature image to obtain a second-level compact feature image, and accordingly obtaining an Nth-level compact feature image, wherein the spatial information of the Nth-level compact feature image is the same as the spatial information of the image to be segmented, and taking the Nth-level compact feature image as a target image to be segmented;
predicting the class to be segmented of each pixel in the target image to be segmented to obtain a segmentation prediction image of the target image to be segmented;
and extracting the prediction boundary of the target image to be segmented according to the segmentation prediction graph.
2. The image segmentation method according to claim 1, wherein extracting the prediction boundary of the target image to be segmented according to the segmentation prediction map comprises:
extracting boundary pixels from a segmentation prediction image of the target image to be segmented, wherein the types of the classes to be segmented of all pixels in a preset neighborhood of the boundary pixels comprise at least two types;
respectively acquiring the number of boundary pixels in each pixel preset neighborhood;
taking the number of boundary pixels in each pixel preset neighborhood as the weight of the corresponding pixel;
calculating the loss of a preset neighborhood of each pixel according to the weight of each pixel;
and according to the loss of the preset neighborhood of each pixel, the model is updated in an iterative mode, and the prediction result of the target image to be segmented is finally obtained.
3. The image segmentation method according to claim 2, wherein the weight of the pixel is assigned to 1 when the number of boundary pixels in the pixel preset neighborhood is 0.
4. The image segmentation method according to any one of claims 1 to 3, wherein the step-by-step encoding of the image to be segmented to obtain the highest-level feature image of the image to be segmented comprises:
the method comprises the steps of coding an image to be segmented step by step to obtain a multilayer characteristic image, wherein spatial information of the multilayer characteristic image is sequentially reduced, and semantic information is sequentially increased;
and acquiring a characteristic image with minimum spatial information and maximum semantic information in the multilayer characteristic images as a highest-level characteristic image of the image to be segmented.
5. An image segmentation apparatus, comprising:
the system comprises an encoding module, a semantic segmentation network and a feature image acquisition module, wherein the encoding module is used for inputting an image to be segmented into an encoding module of the semantic segmentation network to perform step-by-step encoding to obtain a multilayer feature image, the spatial information of the multilayer feature image is sequentially reduced, the semantic information is sequentially increased, and the feature image with the minimum spatial information and the maximum semantic information in the multilayer feature image is acquired as a highest-level feature image;
the input module is used for sending the highest-level feature image to the compact feature module, calculating feature centers of various types to be segmented through rough segmentation prediction results in the compact feature learning module, and then performing feature reconstruction on the highest-level feature image according to the similarity between each pixel in the highest-level feature image and each feature center to obtain a compact feature image;
the decoding module is used for combining the obtained compact feature map with the highest-level feature map to obtain a first-level compact feature image, obtaining a feature image in the encoding process, the spatial information of which is the same as that of the first-level compact feature image, performing feature fusion on the first-level compact feature image and the feature image to obtain a second-level compact feature image, and accordingly obtaining an Nth-level compact feature image, the spatial information of which is the same as that of the image to be segmented, and taking the Nth-level compact feature image as a target image to be segmented;
the prediction module is used for predicting the class to be segmented of each pixel in the target image to be segmented to obtain a segmentation prediction image of the target image to be segmented;
and the extraction module is used for extracting the prediction boundary of the target image to be segmented according to the segmentation prediction graph.
6. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the image segmentation method according to any one of claims 1 to 4 when executing the computer program.
7. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the image segmentation method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010085655.9A CN111340820B (en) | 2020-02-10 | 2020-02-10 | Image segmentation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010085655.9A CN111340820B (en) | 2020-02-10 | 2020-02-10 | Image segmentation method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111340820A CN111340820A (en) | 2020-06-26 |
| CN111340820B true CN111340820B (en) | 2022-05-17 |
Family
ID=71181405
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010085655.9A Active CN111340820B (en) | 2020-02-10 | 2020-02-10 | Image segmentation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111340820B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111915627B (en) * | 2020-08-20 | 2021-04-16 | 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) | Semantic segmentation method, network, device and computer storage medium |
| CN113284155B (en) * | 2021-06-08 | 2023-11-07 | 京东科技信息技术有限公司 | Video object segmentation method and device, storage medium and electronic equipment |
| CN113822901B (en) * | 2021-07-21 | 2023-12-12 | 南京旭锐软件科技有限公司 | Image segmentation method and device, storage medium and electronic equipment |
| CN114565759A (en) * | 2022-02-22 | 2022-05-31 | 北京百度网讯科技有限公司 | Image semantic segmentation model optimization method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109872306A (en) * | 2019-01-28 | 2019-06-11 | 腾讯科技（深圳）有限公司 | Medical image cutting method, device and storage medium |
| CN110148192A (en) * | 2019-04-18 | 2019-08-20 | 上海联影智能医疗科技有限公司 | Medical image imaging method, device, computer equipment and storage medium |
| CN110490203A (en) * | 2019-07-05 | 2019-11-22 | 平安科技（深圳）有限公司 | Image partition method and device, electronic equipment and computer readable storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110838124B (en) * | 2017-09-12 | 2021-06-18 | 深圳科亚医疗科技有限公司 | Method, system, and medium for segmenting images of objects having sparse distribution |
| CN108335313A (en) * | 2018-02-26 | 2018-07-27 | 阿博茨德（北京）科技有限公司 | Image partition method and device |
-
2020
- 2020-02-10 CN CN202010085655.9A patent/CN111340820B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109872306A (en) * | 2019-01-28 | 2019-06-11 | 腾讯科技（深圳）有限公司 | Medical image cutting method, device and storage medium |
| CN110148192A (en) * | 2019-04-18 | 2019-08-20 | 上海联影智能医疗科技有限公司 | Medical image imaging method, device, computer equipment and storage medium |
| CN110490203A (en) * | 2019-07-05 | 2019-11-22 | 平安科技（深圳）有限公司 | Image partition method and device, electronic equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111340820A (en) | 2020-06-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111340820B (en) | Image segmentation method and device, electronic equipment and storage medium | |
| US11222211B2 (en) | Method and apparatus for segmenting video object, electronic device, and storage medium | |
| WO2020098422A1 (en) | Encoded pattern processing method and device , storage medium and electronic device | |
| CN115661144A (en) | Self-adaptive medical image segmentation method based on deformable U-Net | |
| CN113221925B (en) | Target detection method and device based on multi-scale image | |
| WO2018230294A1 (en) | Video processing device, display device, video processing method, and control program | |
| CN112597918A (en) | Text detection method and device, electronic equipment and storage medium | |
| JP2019164618A (en) | Signal processing apparatus, signal processing method and program | |
| CN111709428B (en) | Method and device for identifying positions of key points in image, electronic equipment and medium | |
| CN111145202B (en) | Model generation method, image processing method, device, equipment and storage medium | |
| CN116645592A (en) | Crack detection method based on image processing and storage medium | |
| CN110516598B (en) | Method and apparatus for generating image | |
| Barzigar et al. | A video super-resolution framework using SCoBeP | |
| CN110929731B (en) | Medical image processing method and device based on pathfinder intelligent search algorithm | |
| CN116129417A (en) | Digital instrument reading detection method based on low-quality image | |
| CN116188535A (en) | Video tracking method, device, equipment and storage medium based on optical flow estimation | |
| CN112102216B (en) | Self-adaptive weight total variation image fusion method | |
| Zhao et al. | Single image super-resolution reconstruction using multiple dictionaries and improved iterative back-projection | |
| Azhar et al. | Texture-oriented image denoising technique for the removal of random-valued impulse noise | |
| CN113596576A (en) | Video super-resolution method and device | |
| CN113610856A (en) | Method and device for training image segmentation model and image segmentation | |
| Xu et al. | Discarding jagged artefacts in image upscaling with total variation regularisation | |
| CN117252881B (en) | Bone age prediction method, system, equipment and medium based on hand X-ray image | |
| CN111179246B (en) | Pixel displacement confirming method and device, electronic equipment and storage medium | |
| CN107886522B (en) | Scale-adaptive target model updating method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |