CN111598904B - Image segmentation method, device, equipment and storage medium - Google Patents
Image segmentation method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111598904B CN111598904B CN202010434332.6A CN202010434332A CN111598904B CN 111598904 B CN111598904 B CN 111598904B CN 202010434332 A CN202010434332 A CN 202010434332A CN 111598904 B CN111598904 B CN 111598904B
- Authority
- CN
- China
- Prior art keywords
- organ
- image
- organ image
- feature map
- segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Epidemiology (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application provides an image segmentation method, an image segmentation device, image segmentation equipment and a storage medium, and relates to the technical field of image processing and machine learning. The method comprises the following steps: acquiring a first organ image to be segmented, a second organ image with a label and label information of the second organ image, wherein the label information is used for indicating an organ contained in the second organ image; extracting k single organ images from the second organ image according to the label information to obtain prior information, wherein the prior information is used for indicating the form and the position of the organ contained in the second organ image; extracting a feature map of the second organ image based on the prior information; and according to the characteristic diagram of the first organ image and the characteristic diagram of the second organ image, carrying out similarity analysis on the first organ image and the second organ image to obtain an organ segmentation result of the first organ image. By adopting the technical scheme provided by the embodiment of the application, the accuracy of organ segmentation can be improved.
Description
Technical Field
The embodiment of the application relates to the technical field of image segmentation and machine learning, in particular to an image segmentation method, an image segmentation device, image segmentation equipment and a storage medium.
Background
Medical imaging techniques are widely used in modern medicine. Sometimes, further processing is required for medical images (e.g., organ images) acquired directly.
In the related art, a two-dimensional organ image to be segmented is input into a segmentation model, and the segmentation model processes the organ image according to parameters of the segmentation model to obtain an organ segmentation result of the organ image. The segmentation model is obtained by training a large number of training samples.
In the above-mentioned techniques, only the image of the organ to be segmented is input into the segmentation model, and when the difference between the image of the organ to be segmented and the two-dimensional image of the organ as the training sample is large, the obtained segmentation result of the organ is not accurate enough.
Disclosure of Invention
The embodiment of the application provides an image segmentation method, an image segmentation device, image segmentation equipment and a storage medium, which can improve the accuracy of organ segmentation. The technical scheme is as follows:
in one aspect, an embodiment of the present application provides an image segmentation method, where the method includes:
acquiring a first organ image to be segmented, a second organ image with a label, and label information of the second organ image, wherein the label information is used for indicating an organ contained in the second organ image;
extracting k single organ images from the second organ image according to the label information, wherein the prior information is used for indicating the form and the position of the organ contained in the second organ image, the single organ image refers to the image containing the single organ in the second organ image, and k is a positive integer;
extracting a feature map of the second organ image based on the prior information;
and according to the characteristic diagram of the first organ image and the characteristic diagram of the second organ image, carrying out similarity analysis on the first organ image and the second organ image to obtain an organ segmentation result of the first organ image.
In another aspect, an embodiment of the present application provides an image segmentation apparatus, including:
an image acquisition module, configured to acquire a first organ image to be segmented, a second organ image with a tag, and tag information of the second organ image, where the tag information is used to indicate an organ included in the second organ image;
an information obtaining module, configured to extract k single organ images from the second organ image according to the tag information to obtain prior information, where the prior information is used to indicate a form and a position of an organ included in the second organ image, the single organ image is an image including a single organ in the second organ image, and k is a positive integer;
and the result acquisition module is used for extracting the feature map of the second organ image based on the prior information, and performing similarity analysis on the first organ image and the second organ image according to the feature map of the first organ image and the feature map of the second organ image to obtain an organ segmentation result of the first organ image.
In yet another aspect, an embodiment of the present application provides a computer device, which includes a processor and a memory, where at least one instruction, at least one program, a code set, or a set of instructions is stored in the memory, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by the processor to implement the above-mentioned image segmentation method.
In yet another aspect, the present application provides a computer-readable storage medium, where at least one instruction, at least one program, a code set, or a set of instructions is stored, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by a processor to implement the above-mentioned image segmentation method.
In yet another aspect, the present application provides a computer program product, which is executed by a processor to implement the image segmentation method.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
the method comprises the steps of obtaining a first organ image to be segmented and a second organ image with a label, extracting k single organ images from the second organ image with the label to obtain prior information of a second organ, extracting a feature map of the second organ image based on the prior information, and performing similarity analysis on the first organ image and the second organ image according to the feature map of the first organ image and the feature map of the second organ image to obtain an organ segmentation result of the first organ image.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a flow chart of an image segmentation method provided by an embodiment of the present application;
FIG. 2 is a schematic diagram of an organ segmentation model provided by an embodiment of the present application;
FIG. 3 is a flow chart of an image segmentation method provided by another embodiment of the present application;
FIG. 4 is a schematic diagram of a similarity calculation module provided in one embodiment of the present application;
FIG. 5 is a schematic diagram of an image segmentation system provided by an embodiment of the present application;
FIG. 6 is a block diagram of an image segmentation apparatus provided in one embodiment of the present application;
fig. 7 is a block diagram of an image segmentation apparatus according to another embodiment of the present application;
FIG. 8 is a block diagram of a computer device provided by one embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of methods consistent with aspects of the present application, as detailed in the appended claims.
Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the implementation method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.
The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.
Computer Vision technology (Computer Vision, CV): computer vision is a science for researching how to make a machine "see", and further, it means that a camera and a computer are used to replace human eyes to perform machine vision such as identification, tracking and measurement on a target, and further image processing is performed, so that the computer processing becomes an image more suitable for human eyes to observe or transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. Computer vision technologies generally include image segmentation, image Recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content/behavior Recognition, three-dimensional object reconstruction, 3D (three-dimensional) technology, virtual reality, augmented reality, synchronous positioning, map construction, and other technologies, and also include common biometric technologies such as face Recognition and fingerprint Recognition.
Machine Learning (ML) is a multi-domain cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
With the research and progress of artificial intelligence technology, the artificial intelligence technology is developed and applied in a plurality of fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, unmanned aerial vehicles, robots, smart medical care, smart customer service, and the like.
The scheme provided by the embodiment of the application relates to the technical field of image segmentation and machine learning, and the following embodiment is specifically used for carrying out image segmentation on an organ image by utilizing the machine learning technology to obtain an organ segmentation result of the organ image.
First, before further detailed description of the embodiments of the present application, terms and expressions referred to in the embodiments of the present application will be explained.
1. Prior information: refers to empirical and historical information that has been obtained by a computer device prior to processing a sample.
2. Encoder network (Encoder): and the module is used for converting the image with the original resolution into the feature map.
3. Decoder network (Decoder): and converting the characteristic diagram into an image with the original resolution.
4. A similarity calculation module: and the module is used for carrying out similarity calculation.
5. Feature map (feature map): the output of various levels in the encoder or decoder.
According to the method provided by the embodiment of the application, the execution main body of each step can be a computer device, and the computer device refers to an electronic device with data calculation, processing and storage capabilities. The Computer device may be a terminal such as a PC (Personal Computer), tablet, smartphone, wearable device, smart robot, or the like; or may be a server. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services. The technical solution of the present application will be described below by means of several embodiments.
Referring to fig. 1, a flowchart of an image segmentation method according to an embodiment of the present application is shown. The method comprises the following steps (110-140):
The organ image is an image for representing the structure, state, and relative positional relationship of organs in a living body, and the organ image includes at least one organ. Organs include eyes, ears, nose, tongue, heart, liver, lungs, stomach, kidneys, bones, skin, and the like. In some examples, the first organ image and the second organ image are three-dimensional organ images; in other examples, the first organ image and the second organ image are two-dimensional organ images. Optionally, the organ image comprises: CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, B-scan ultrasound (B-scan ultrasound) images, and the like, which are not limited in the embodiments of the present application. It should be noted that the first organ image and the second organ image are the same type of image (e.g., the first organ image and the second organ image are both CT images, or are both MRI images, or are both B-mode ultrasound images). The first organ image and the second organ image are similar images. Optionally, the first organ image and the second organ image are images taken of the same region of the same type of living being. For example, the first organ image and the second organ image are chest organ images of the same patient; for another example, the first organ image and the second organ image are chest organ images of different patients. The first organ image is an organ image to be segmented, and the organ segmentation result of the first organ image can be obtained by the image segmentation method of the embodiment of the application.
In an embodiment of the application, the second organ image has label information indicating an organ contained in the second organ image. For example, the tag information includes the size, name, number, nature, type, and the like of the organ included in the second organ image.
And step 120, acquiring prior information from the second organ image according to the label information.
Optionally, a priori information is used to indicate the morphology and location of the organ contained in the second organ image. The morphology of the organ refers to the contour shape of the organ, the color and brightness of each region in the organ, and the position of the organ refers to the relative position between each organ and the position of each organ in the organ image. In some embodiments, the second organ image is subjected to organ segmentation based on the label information, resulting in prior information.
In some embodiments, k single organ images are extracted from the second organ image according to the tag information, and the k single organ images and the second organ image are merged in the channel dimension to obtain a processed second organ image (i.e., the processed second organ image includes the k single organ images and the second organ image). The single organ image is an image including a single organ in the second organ image, and k is a positive integer. The prior information includes k single organ images. According to the label information, k single organ images are extracted from the second organ image, and each single organ image only comprises one organ. In one example, the second organ image is a chest cavity image of a human body, the organs in the second organ image include a heart, a lung, and ribs, and the k single organ images include a heart image, a lung image, and a rib image.
The processed second organ image is obtained by splicing the k single organ images and the second organ image in the channel dimension, so that the feature extraction is performed on the processed second organ image, and the feature map of the second organ image based on the prior information can be obtained.
Similarity analysis results of the first organ image and the second organ image can be obtained by calculating the similarity of the feature map of the first organ image and the feature map of the second organ image. The analysis result of the similarity analysis is combined with the feature map of the first organ image, so that the organ segmentation result of the first organ image can be obtained.
In some embodiments, feature extraction results of the first organ image and the processed second organ image (e.g., the feature map of the first organ image and the feature map of the second organ image) are obtained by performing feature extraction processing on the first organ image and the processed second organ image based on the prior information. And performing similarity analysis on the feature extraction results of the first organ image and the processed second organ image to obtain a similarity analysis result. And combining the analysis result of the similarity analysis with the feature extraction result of the first organ image to obtain an organ segmentation result of the first organ image. Optionally, the organ segmentation result of the first organ image is an organ mask corresponding to the first organ image. In the organ mask corresponding to the first organ image, the encoding of the pixel points of the same organ is the same.
In summary, in the technical solution provided in the embodiment of the present application, by obtaining a first organ image to be segmented and a labeled second organ image, k single organ images are extracted from the labeled second organ image to obtain prior information of a second organ, a feature map of the second organ image is extracted based on the prior information, and then, according to the feature map of the first organ image and the feature map of the second organ image, a similarity analysis is performed on the first organ image and the second organ image to obtain an organ segmentation result of the first organ image.
In the technical scheme provided by the embodiment of the application, the first organ image and the second organ image are three-dimensional organ images, correspondingly, the organ segmentation result of the obtained first organ image is also three-dimensional, and compared with the segmentation result of the two-dimensional organ image, the organ segmentation result of the first organ image is more intuitive and closer to the actual situation, and the convenience of obtaining organ information is improved.
In some embodiments, the organ segmentation result of the first organ image is generated by an organ segmentation model. As shown in fig. 2, the organ segmentation model 20 includes a primary encoder network 21, a secondary encoder network 22, a similarity calculation module 23, and a decoder network 24. Wherein, the first organ image 25 is inputted into the main encoder network 21; the processed second organ image 26 is input to the auxiliary encoder network 22; the similarity calculation module 23 is configured to perform similarity analysis on intermediate output information of the primary encoder network 21 and the secondary encoder network 22; the decoder network 24 outputs the organ segmentation result 27. Alternatively, the similarity calculation module 23 includes n similarity calculation units.
Referring to fig. 3, a flowchart of an image segmentation method according to another embodiment of the present application is shown. As shown in FIG. 3, the steps 130-140 can be alternatively realized by the following steps (131-134):
In some embodiments, the primary encoder network includes a plurality of cascaded feature extraction layers for down-sampling a plurality of times based on the first organ image. Each feature extraction layer of the main encoder network includes a plurality of basic blocks, such as convolution, BN (Batch Normalization), ReLU (Rectified Linear Units layer), max-pooling layer, and so on. A basic block in a feature extraction layer of a main encoder network can perform down-sampling once based on a first organ image, the obtained output resolution is 1/L of the originally input feature extraction layer, and L is larger than 1. Optionally, L is 2. And the main encoder performs down-sampling on the last feature extraction layer to obtain output, namely the feature map of the first organ image. If the main encoder network has performed 3 downsampling operations, and L for each downsampling is 2, the resolution of the feature map of the first organ image is 1/8 of the original input (i.e., first organ image) resolution.
In some embodiments, the primary and secondary encoder networks are identical in structure. The input to the secondary encoder network is the processed second organ image. The secondary encoder network includes a plurality of cascaded feature extraction layers for down-sampling a plurality of times based on the prior information and the processed second organ image. Each feature extraction layer of the secondary encoder network includes a plurality of basic blocks such as convolution, BN, ReLU, max-pooling layers, and so on. Through each basic block in a feature extraction layer of the auxiliary encoder network, downsampling can be performed for one time based on prior information and the processed second organ image, the obtained output resolution is 1/L of the original input feature extraction layer, and L is larger than 1. Optionally, the output obtained by the secondary encoder through the last downsampling is the feature map of the second organ image. Optionally, the feature map of the first organ image and the feature map of the second organ image have equal resolution.
The similarity calculation module can calculate the corresponding similarity based on the output of the feature extraction layers of the primary encoder network and the secondary encoder network. When the feature map of the first organ image and the feature map of the second organ image are input into the similarity calculation module, the similarity calculation module can obtain the target feature map of the first organ image by calculating the similarity between the feature map of the first organ image and the feature map of the second organ image.
In some embodiments, as shown in fig. 4, this step 133 includes the sub-steps of:
1. similarity calculation is carried out on the feature map 41 of the first organ image and the feature map 42 of the second organ image through a similarity calculation module 23 to obtain an attention weight map 43;
2. the feature map 41 of the first organ image is multiplied by the attention weight map 43 to obtain a target feature map 44 corresponding to the first organ image.
The similarity calculation module 23 can calculate the similarity between the feature map 41 of the first organ image and the feature map 42 of the second organ image, and the calculation result of the obtained similarity is the attention weight map 43. The first organ image and the second organ image are both three-dimensional organ images. The feature map 41 of the first organ image, the feature map 42 of the second organ image and the target feature map 44 each comprise four dimensions: d (Depth), H (Height), W (Width), and C (Channel dimension), where the D dimension is not shown in fig. 4. Attention weight graph 43 includes three dimensions: H. w and C. The feature map 41 of the first organ image is multiplied by the attention weight map 43, and the obtained feature map is the target feature map 44.
Optionally, the Similarity calculation Module adopts Cosine Similarity calculation, and the Similarity calculation Module is a Cosine Similarity calculation Module (CSAM), and the Cosine Similarity calculation method refers to the following formula:
wherein f is 1 A profile representing the output in the primary encoder network, f 2 A feature map representing an output in the secondary encoder network; the vectors i, j and k respectively represent H, W, C dimension corresponding numbers; w i，j，k Representing a vector weight in the attention weight map;
represent
The L2 norm of (a),
to represent
L2 norm.
In some embodiments, after the target feature map is obtained by the similarity calculation module, the decoder network performs processing such as upsampling on the target feature map, so that the resolution of the target feature map is converted into the resolution of the first organ image, and further the organ segmentation result of the first organ image is generated. Optionally, the decoder network comprises a plurality of basic blocks, such as convolutional networks, BN, activation functions, and so on. The method for implementing upsampling includes deconvolution, interpolation, and the like, which is not limited in the embodiments of the present application.
In some embodiments, this step 134 includes the following sub-steps:
1. performing up-sampling processing on the target characteristic diagram through a decoder network to obtain an up-sampled characteristic diagram;
2. performing dimensionality reduction and normalization processing on the up-sampled feature map to obtain a processed feature map; the pixel value of the ith pixel point in the processed characteristic diagram is used for indicating the probability that the ith pixel point belongs to each organ, and i is a positive integer;
3. based on the processed feature map, an organ segmentation result of the first organ image is generated.
In some embodiments, the target feature map is upsampled by the decoder network the same number of times as the downsampling by the main encoder network. And the characteristic diagram obtained by the last upsampling of the decoder network is the upsampled characteristic diagram. And (3) reducing the dimension of the up-sampled feature map by a convolution network (such as a 1 × 1 × 1 convolution network), and performing normalization processing (such as normalization processing by a softmax function) on the feature map subjected to dimension reduction by the convolution network to obtain a processed feature map.
In some embodiments, the processed feature map includes a plurality of pixel points. The pixel value of the ith pixel point in the processed feature map includes (k +1) probability values corresponding to the ith pixel point. The (k +1) probability values include k probability values corresponding to the k organs one-to-one, and 1 probability value corresponding to the background region. And the mth probability value in the k probability values is used for indicating the probability that the ith pixel point belongs to the mth organ in the k organs. And 1 probability value is used for indicating the probability that the ith pixel point belongs to the background area. k is a positive integer, and m is a positive integer less than or equal to k. If the maximum value of the (k +1) probability values corresponds to the first organ, determining the ith pixel point as the pixel point of the first organ; and if the maximum value of the (k +1) probability values corresponds to the background area, determining the ith pixel point as the pixel point of the background area. And respectively executing the steps on a plurality of pixel points in the processed characteristic diagram to obtain the segmentation result of the first organ image.
In one example, the processed feature map includes 3 organs, respectively: for the pixel 1 in the processed feature map, the probability value corresponding to the organ a is 0.02, the probability value corresponding to the organ B is 0.9, the probability value corresponding to the organ C is 0.05, and the probability value corresponding to the background region is 0.03. Since 0.9>0.05>0.03>0.02, the pixel point 1 is the pixel point of the organ B corresponding to the probability value 0.9.
In summary, in the technical solution provided in the embodiment of the present application, similarity calculation is performed on the feature map of the first organ image and the feature map of the second organ image based on the prior information, and the similarity calculation result is fused into the processing procedure of organ segmentation on the first organ image, so that the accuracy of organ segmentation is improved. Therefore, only a small amount of labeled second organ images are needed, a relatively accurate organ segmentation result of the first organ image can be obtained, and the organ segmentation efficiency of the first organ image is improved.
In some embodiments, the training process of the organ segmentation model is as follows:
1. selecting a training sample from a training set, wherein the training sample comprises a first organ image sample and a second organ image sample;
2. acquiring prior information from the second organ image sample according to the label information of the second organ image sample;
3. performing similarity analysis on the first organ image sample and the second organ image sample through an organ segmentation model based on prior information of the second organ image sample to obtain an organ segmentation result of the first organ image sample;
4. calculating a loss function value corresponding to the organ segmentation model according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample;
5. and adjusting the parameters of the organ segmentation model according to the loss function values.
Optionally, the training set comprises a plurality of samples for training the organ segmentation model. In the training process of the organ segmentation model, the contents of each step of obtaining the organ segmentation result of the first organ image sample may refer to the contents of the embodiment in fig. 1, and are not described herein again. The label information of the first organ image sample is: an example of organ segmentation of the first organ image sample obtained by other means (i.e. without using the organ segmentation model in the embodiment of the present application). And calculating a loss function value corresponding to the organ segmentation model through the loss function according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample.
In some embodiments, calculating a corresponding loss function value of the organ segmentation model according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample includes:
1. calculating the similarity between two contour regions of the same organ according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample;
2. and determining a loss function value corresponding to the organ segmentation model based on the similarity.
Alternatively, the more similarity between two contour regions for the same organ, the smaller the value of the corresponding loss function for that organ. And determining the average value of the loss function values corresponding to the organs in the first organ image sample as the loss function value corresponding to the organ segmentation model. Accordingly, the more similarity between the label information of the first organ image sample and the organ segmentation result of the first organ image sample, the smaller the corresponding loss function value of the organ segmentation model. The organ segmentation model tends to output an organ segmentation result with a smaller loss function value by adjusting its own parameters.
In some embodiments, calculating a loss function value corresponding to the organ segmentation model according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample includes the following steps:
1. for the x organ in the first organ image sample, acquiring a first organ area point set used for representing the x organ in the organ segmentation result of the first organ image sample, and using a second organ area point set used for representing the x organ in the label information of the first organ image sample;
2. subtracting the first ratio of two times from 1 to obtain a loss function value corresponding to the xth organ, wherein the first ratio is the ratio of a first base number and a second base number, the first base number is the base number of the intersection of the first organ region point set and the second organ region point set, and the second base number is the sum of the base number of the first organ region point set and the base number of the second organ region point set;
3. and determining the average value of the loss function values corresponding to the organs in the first organ image sample as the loss function value corresponding to the organ segmentation model.
Optionally, calculating the loss function value Dice (a, B) corresponding to the organ segmentation model refers to the following formula two:
where A represents a first organ region point set and B represents a second organ region point set.
In the implementation manner, the loss function value corresponding to the organ segmentation model is calculated, so that the organ segmentation model adjusts parameters thereof according to the loss function value, and the organ segmentation model tends to output an organ segmentation result with a smaller loss function value, that is, the organ segmentation model tends to output an organ segmentation result with a higher similarity of label information of the first organ image sample, so as to obtain a more optimized organ segmentation model, and improve the training efficiency of the organ segmentation model.
In some embodiments, the trained organ segmentation model is tested, and the testing process comprises the following steps:
1. selecting a reference organ image sample (template image) from the training set, and acquiring label information of the reference organ image sample;
2. selecting a target organ image sample (target image) from the test set, and acquiring label information of the target organ image sample;
3. extracting single organ images of organs in the reference organ images according to the label information of the reference organ image samples, and splicing each single organ image of the reference organ images with the reference organ images to obtain processed reference organ image samples;
4. inputting the target organ image sample and the processed reference organ image sample into an organ segmentation model to obtain an organ segmentation result of the target organ image sample;
4. and analyzing the accuracy of the organ segmentation result by comparing the organ segmentation result of the target organ image sample with the label information of the target organ image sample. If the accuracy of the organ segmentation result meets the preset condition, the organ segmentation model training is finished; and if the accuracy of the organ segmentation result does not meet the preset condition, continuing to train the organ segmentation model.
Referring to fig. 5, a schematic diagram of an image segmentation system according to an embodiment of the present application is shown. As shown in fig. 5, the system 50 includes a first terminal 51, a backend server 52, and a second terminal 53. The first terminal 51 sends the organ image with the label and the organ image without the label to the backend server 52, and the backend server 52 obtains the organ segmentation result of the organ image without the label by using the image segmentation method provided by the embodiment of the application, and sends the organ segmentation result of the organ image without the label to the second terminal 53. Optionally, the unlabeled organ image includes multiple organ images. In one example, the first terminal 51 and the second terminal 53 are the same terminal; in another example, the first terminal 51 and the second terminal 53 are different terminals, which is not limited in this embodiment of the present application.
The following are embodiments of the apparatus of the present application that may be used to perform embodiments of the method of the present application. For details which are not disclosed in the embodiments of the apparatus of the present application, reference is made to the embodiments of the method of the present application.
Referring to fig. 6, a block diagram of an image segmentation apparatus according to an embodiment of the present application is shown. The device has the functions of implementing the method examples of image segmentation, and the functions can be realized by hardware or by hardware executing corresponding software. The apparatus may be the computer device described above, or may be provided on a computer device. The apparatus 600 may comprise: an image acquisition module 610, an information acquisition module 620, and a result acquisition module 630.
The image obtaining module 610 is configured to obtain a first organ image to be segmented, a second organ image with a tag, and tag information of the second organ image, where the tag information is used to indicate an organ included in the second organ image.
The information obtaining module 620 is configured to extract k single organ images from the second organ image according to the tag information to obtain prior information, where the prior information is used to indicate a form and a position of an organ included in the second organ image, the single organ image is an image including a single organ in the second organ image, and k is a positive integer.
The result obtaining module 630 is configured to perform similarity analysis on the first organ image and the second organ image according to the feature map of the first organ image and the feature map of the second organ image based on the feature information of the second organ image extracted by the priori information, so as to obtain an organ segmentation result of the first organ image.
In summary, in the technical solution provided in the embodiment of the present application, by obtaining a first organ image to be segmented and a labeled second organ image, extracting k single organ images from the labeled second organ image to obtain prior information of a second organ, extracting a feature map of the second organ image based on the prior information, and then performing similarity analysis on the first organ image and the second organ image according to the feature map of the first organ image and the feature map of the second organ image to obtain an organ segmentation result of the first organ image.
In some embodiments, the organ segmentation result of the first organ image is generated by an organ segmentation model comprising a primary encoder network, a secondary encoder network, a similarity calculation module, and a decoder network. As shown in fig. 7, the result obtaining module 630 includes: a feature map extraction sub-module 631, a feature map generation sub-module 632, and a result acquisition sub-module 633.
The feature map extracting sub-module 631 is configured to extract a feature map of the first organ image through the primary encoder network.
The feature map extracting sub-module 631 is further configured to extract, through the auxiliary encoder network, a feature map of the second organ image based on the prior information.
The feature map generation submodule 632 is configured to generate, by the similarity calculation module, a target feature map corresponding to the first organ image based on the similarity between the feature map of the first organ image and the feature map of the second organ image.
The result obtaining sub-module 633 is configured to obtain an organ segmentation result of the first organ image based on the target feature map through the decoder network.
In some embodiments, as shown in fig. 7, the feature map generation module 632 is configured to:
calculating the similarity of the feature map of the first organ image and the feature map of the second organ image through the similarity calculation module to obtain an attention weight map;
and multiplying the feature map of the first organ image and the attention weight map to obtain a target feature map corresponding to the first organ image.
In some embodiments, as shown in fig. 7, the result obtaining sub-module 633 is configured to:
performing upsampling processing on the target characteristic diagram through the decoder network to obtain an upsampled characteristic diagram;
performing dimensionality reduction and normalization processing on the up-sampled feature map to obtain a processed feature map; the pixel value of the ith pixel point in the processed feature map is used for indicating the probability that the ith pixel point belongs to each organ, and i is a positive integer;
and generating an organ segmentation result of the first organ image based on the processed feature map.
In some embodiments, as shown in fig. 7, the apparatus 600 further comprises: a sample selection module 640, a loss calculation module 650, and a parameter adjustment module 660.
The sample selecting module 640 is configured to select a training sample from a training set, where the training sample includes a first organ image sample and a second organ image sample.
The information obtaining module 620 is further configured to obtain prior information from the second organ image sample according to the label information of the second organ image sample.
The result obtaining module 630 is further configured to perform similarity analysis on the first organ image sample and the second organ image sample based on prior information of the second organ image sample through the organ segmentation model, so as to obtain an organ segmentation result of the first organ image sample.
The loss calculation module 650 is configured to calculate a loss function value corresponding to the organ segmentation model according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample;
the parameter adjusting module 660 is configured to adjust a parameter of the organ segmentation model according to the loss function value.
In some embodiments, as shown in fig. 7, the loss calculation module 650 is configured to:
calculating the similarity between two contour regions of the same organ according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample;
and determining a loss function value corresponding to the organ segmentation model based on the similarity.
In some embodiments, the first organ image and the second organ image are three-dimensional organ images.
It should be noted that, when the apparatus provided in the foregoing embodiment implements the functions thereof, only the division of the functional modules is illustrated, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the apparatus may be divided into different functional modules to implement all or part of the functions described above. In addition, the apparatus and method embodiments provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments for details, which are not described herein again.
Referring to fig. 8, a block diagram of a computer device according to an embodiment of the present application is shown. The computer device is used for implementing the image segmentation method provided in the above embodiments. Specifically, the method comprises the following steps:
the computer apparatus 800 includes a CPU (Central Processing Unit) 801, a system Memory 804 including a RAM (Random Access Memory) 802 and a ROM (Read-Only Memory) 803, and a system bus 805 connecting the system Memory 804 and the Central Processing Unit 801. The computer device 800 also includes a basic I/O (Input/Output) system 806 for facilitating information transfer between devices within the computer, and a mass storage device 807 for storing an operating system 813, application programs 814, and other program modules 812.
The basic input/output system 806 includes a display 808 for displaying information and an input device 809 such as a mouse, keyboard, etc. for user input of information. Wherein the display 808 and the input device 809 are connected to the central processing unit 801 through an input output controller 810 connected to the system bus 805. The basic input/output system 806 may also include an input/output controller 810 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, input-output controller 810 also provides output to a display screen, a printer, or other type of output device.
The mass storage device 807 is connected to the central processing unit 801 through a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable media provide non-volatile storage for the computer device 800. That is, the mass storage device 807 may include a computer-readable medium (not shown) such as a hard disk or CD-ROM (Compact disk Read-Only Memory) drive.
Without loss of generality, the computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash Memory or other solid state Memory technology, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the foregoing. The system memory 804 and mass storage 807 described above may be collectively referred to as memory.
According to various embodiments of the present application, the computer device 800 may also operate as a remote computer connected to a network via a network, such as the Internet. That is, the computer device 800 may be connected to the network 812 through the network interface unit 811 coupled to the system bus 805, or may be connected to other types of networks or remote computer systems (not shown) using the network interface unit 811.
In some embodiments, there is also provided a computer readable storage medium having stored therein at least one instruction, at least one program, set of codes, or set of instructions which, when executed by a processor, implement the above-described image segmentation method.
In some embodiments, a computer program product is also provided for implementing the image segmentation method described above when executed by a processor.
It should be understood that reference to "a plurality" herein means two or more. Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It will be understood that the present application is not limited to the precise arrangements that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (10)
1. A method of image segmentation, the method comprising:
acquiring a first organ image to be segmented, a second organ image with a label, and label information of the second organ image, wherein the label information is used for indicating an organ contained in the second organ image;
extracting k single organ images from the second organ image according to the label information to obtain prior information, wherein the prior information is used for indicating the form and the position of the organ contained in the second organ image, the single organ image refers to the image containing the single organ in the second organ image, and k is a positive integer;
extracting a feature map of the first organ image, and extracting a feature map of the second organ image based on the prior information;
generating a target feature map corresponding to the first organ image based on the similarity between the feature map of the first organ image and the feature map of the second organ image;
and obtaining an organ segmentation result of the first organ image based on the target feature map.
2. The method of claim 1, wherein the organ segmentation result of the first organ image is generated by an organ segmentation model comprising a primary encoder network, a secondary encoder network, a similarity calculation module, and a decoder network;
the extracting the feature map of the first organ image, extracting the feature map of the second organ image based on the prior information, generating a target feature map corresponding to the first organ image based on the similarity between the feature map of the first organ image and the feature map of the second organ image, and obtaining the organ segmentation result of the first organ image based on the target feature map includes:
extracting a feature map of the first organ image through the primary encoder network;
extracting a feature map of the second organ image based on the prior information through the auxiliary encoder network;
generating a target feature map corresponding to the first organ image based on the similarity between the feature map of the first organ image and the feature map of the second organ image through the similarity calculation module;
and obtaining an organ segmentation result of the first organ image based on the target feature map through the decoder network.
3. The method according to claim 2, wherein the generating, by the similarity calculation module, a target feature map corresponding to the first organ image based on the similarity between the feature map of the first organ image and the feature map of the second organ image comprises:
calculating the similarity of the feature map of the first organ image and the feature map of the second organ image through the similarity calculation module to obtain an attention weight map;
and multiplying the feature map of the first organ image and the attention weight map to obtain a target feature map corresponding to the first organ image.
4. The method of claim 3, wherein obtaining, by the decoder network, an organ segmentation result of the first organ image based on the target feature map comprises:
performing upsampling processing on the target characteristic diagram through the decoder network to obtain an upsampled characteristic diagram;
performing dimensionality reduction and normalization processing on the up-sampled feature map to obtain a processed feature map; the pixel value of the ith pixel point in the processed feature map is used for indicating the probability that the ith pixel point belongs to each organ, and i is a positive integer;
and generating an organ segmentation result of the first organ image based on the processed feature map.
5. The method of claim 2, wherein the organ segmentation model is trained as follows:
selecting a training sample from a training set, wherein the training sample comprises a first organ image sample and a second organ image sample;
acquiring prior information from the second organ image sample according to the label information of the second organ image sample;
performing similarity analysis on the first organ image sample and the second organ image sample through the organ segmentation model based on prior information of the second organ image sample to obtain an organ segmentation result of the first organ image sample;
calculating a loss function value corresponding to the organ segmentation model according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample;
and adjusting the parameters of the organ segmentation model according to the loss function value.
6. The method of claim 5, wherein the calculating a loss function value corresponding to the organ segmentation model according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample comprises:
calculating the similarity between two contour regions of the same organ according to the label information of the first organ image sample and the organ segmentation result of the first organ image sample;
and determining a loss function value corresponding to the organ segmentation model based on the similarity.
7. The method of any one of claims 1 to 6, wherein the first organ image and the second organ image are three-dimensional organ images.
8. An image segmentation apparatus, characterized in that the apparatus comprises:
an image acquisition module, configured to acquire a first organ image to be segmented, a second organ image with a tag, and tag information of the second organ image, where the tag information is used to indicate an organ included in the second organ image;
an information obtaining module, configured to extract k single organ images from the second organ image according to the tag information to obtain prior information, where the prior information is used to indicate a form and a position of an organ included in the second organ image, the single organ image is an image including a single organ in the second organ image, and k is a positive integer;
the result acquisition module is used for extracting a feature map of the first organ image and extracting a feature map of the second organ image based on the prior information; generating a target feature map corresponding to the first organ image based on the similarity between the feature map of the first organ image and the feature map of the second organ image; and obtaining an organ segmentation result of the first organ image based on the target feature map.
9. A computer device comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by the processor to implement the image segmentation method according to any one of claims 1 to 7.
10. A computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the image segmentation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010434332.6A CN111598904B (en) | 2020-05-21 | 2020-05-21 | Image segmentation method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010434332.6A CN111598904B (en) | 2020-05-21 | 2020-05-21 | Image segmentation method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111598904A CN111598904A (en) | 2020-08-28 |
| CN111598904B true CN111598904B (en) | 2022-09-16 |
Family
ID=72192375
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010434332.6A Active CN111598904B (en) | 2020-05-21 | 2020-05-21 | Image segmentation method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111598904B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114693830B (en) * | 2022-05-27 | 2022-11-15 | 阿里巴巴达摩院(杭州)科技有限公司 | Multi-organ segmentation and model training method, equipment and medium for medical image |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109559295A (en) * | 2018-06-04 | 2019-04-02 | 新影智能科技(昆山)有限公司 | Image analysis system, method, computer readable storage medium and electric terminal |
| CN110232691A (en) * | 2019-04-18 | 2019-09-13 | 浙江大学山东工业技术研究院 | A kind of dividing method of multi-modal CT images |
| CN110490881A (en) * | 2019-08-19 | 2019-11-22 | 腾讯科技（深圳）有限公司 | Medical image dividing method, device, computer equipment and readable storage medium storing program for executing |
| CN110599491B (en) * | 2019-09-04 | 2024-04-12 | 腾讯医疗健康(深圳)有限公司 | Priori information-based eye image segmentation method, apparatus, device and medium |
-
2020
- 2020-05-21 CN CN202010434332.6A patent/CN111598904B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111598904A (en) | 2020-08-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11861829B2 (en) | Deep learning based medical image detection method and related device | |
| CN111126272B (en) | Posture acquisition method, and training method and device of key point coordinate positioning model | |
| US11887311B2 (en) | Method and apparatus for segmenting a medical image, and storage medium | |
| CN113077471B (en) | Medical image segmentation method based on U-shaped network | |
| CN111369576B (en) | Training method of image segmentation model, image segmentation method, device and equipment | |
| EP3511942A2 (en) | Cross-domain image analysis and cross-domain image synthesis using deep image-to-image networks and adversarial networks | |
| CN111429460B (en) | Image segmentation method, image segmentation model training method, device and storage medium | |
| KR20210048523A (en) | Image processing method, apparatus, electronic device and computer-readable storage medium | |
| US10853409B2 (en) | Systems and methods for image search | |
| US20230072627A1 (en) | Gaze correction method and apparatus for face image, device, computer-readable storage medium, and computer program product face image | |
| CN111932529B (en) | Image classification and segmentation method, device and system | |
| CN113822851A (en) | Image segmentation method, device, equipment and storage medium | |
| CN113822289A (en) | Training method, device and equipment of image noise reduction model and storage medium | |
| CN113822792A (en) | Image registration method, device, equipment and storage medium | |
| CN113822323A (en) | Brain scanning image identification processing method, device, equipment and storage medium | |
| CN111598904B (en) | Image segmentation method, device, equipment and storage medium | |
| CN114283152A (en) | Image processing method, image processing model training method, image processing device, image processing equipment and image processing medium | |
| CN113592769A (en) | Abnormal image detection method, abnormal image model training method, abnormal image detection device, abnormal image model training device and abnormal image model training medium | |
| CN113822845A (en) | Method, apparatus, device and medium for hierarchical segmentation of tissue structure in medical image | |
| WO2023160157A1 (en) | Three-dimensional medical image recognition method and apparatus, and device, storage medium and product | |
| WO2023207416A1 (en) | Image completion method and apparatus, device, and storage medium | |
| CN115965785A (en) | Image segmentation method, device, equipment, program product and medium | |
| CN111369564B (en) | Image processing method, model training method and model training device | |
| CN113822846A (en) | Method, apparatus, device and medium for determining region of interest in medical image | |
| CN115115772A (en) | Key structure reconstruction method and device based on three-dimensional image and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40027936 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |