CN114549834A - Semi-supervised hybrid training semantic segmentation method and system - Google Patents

Semi-supervised hybrid training semantic segmentation method and system Download PDF

Info

Publication number
CN114549834A
CN114549834A CN202210094172.4A CN202210094172A CN114549834A CN 114549834 A CN114549834 A CN 114549834A CN 202210094172 A CN202210094172 A CN 202210094172A CN 114549834 A CN114549834 A CN 114549834A
Authority
CN
China
Prior art keywords
segmentation
segmentation model
training
convex
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210094172.4A
Other languages
Chinese (zh)
Inventor
蒋朝辉
刘金狮
曹婷
何瑞清
余金花
桂卫华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Central South University
Original Assignee
Central South University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Central South University filed Critical Central South University
Priority to CN202210094172.4A priority Critical patent/CN114549834A/en
Publication of CN114549834A publication Critical patent/CN114549834A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2155—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a semi-supervised hybrid training semantic segmentation method and system, which are used for acquiring a high-overlapping particle image RGB data set, labeling the particle image RGB data set to obtain labeled data and unlabelled data, establishing a pre-segmentation model based on a segmentation algorithm of an encoding and decoding frame, training the pre-segmentation model by adopting the labeled data to obtain a rough segmentation model, identifying the unlabelled data by adopting the rough segmentation model to obtain a rough segmentation result, obtaining a pseudo label based on the rough segmentation result, training the rough segmentation model based on the pseudo label and the labeled data to obtain a fine segmentation model and perform image segmentation on the high-overlapping particle image based on the fine segmentation model, solving the technical problem of low segmentation precision of the existing stacked particle image, realizing high-precision segmentation of the stacked particle image by adopting a pseudo label generation algorithm based on a convex hull, and the structural integrity of the particles can be greatly improved, and the degree of under-segmentation is reduced.

Description

Semi-supervised hybrid training semantic segmentation method and system
Technical Field
The invention mainly relates to the technical field of blast furnace image processing, in particular to a semi-supervised hybrid training semantic segmentation method and system.
Background
Particles generally refer to fine objects, being a finely divided mass that is relatively fragmented, broadly to solid particles or droplets or bubbles having a solid shape. The particles have three most important properties of particle size, number and shape, and the particle size refers to the physical dimension of a single particle, namely an index describing the size of a single particle, and is as large as a centimeter and as small as a nanometer. The number describes the number of particles contained in the population of particles, as the particles are usually present in the form of a population of particles. The shape is an apparent shape representing an individual particle, and is generally classified into a circular shape and a non-circular shape (irregular shape). There is a lot of research on particle analysis in various industries, such as ore particle size detection in industry, cell image segmentation in medicine, leaf segmentation in agriculture, etc. The most widely used method is particle analysis by image segmentation.
The concepts of under-segmentation and over-segmentation exist in image segmentation, and are used for describing the difference characteristics between a segmentation result and a label. In the segmentation process of the grain image, the accuracy of segmentation is more affected by under-segmentation than by over-segmentation. In most over-segmentation situations, excessive pixel points often cannot form a new closed connected domain, and only branches with different lengths are formed, so that the influence on the number of the identified particles, the size of the calculated particles and the positions of the positioned particles can be almost ignored, and only the edge and the shape of the identified particles are partially influenced.
Under most of the under-segmentation conditions, the lack of partial pixels has a fatal influence on an originally closed connected domain, and the lack of some key pixels directly leads to the fusion of a plurality of originally closed connected domains, which has a huge influence on the number calculation, size calculation, position locking, edges and shapes of particles, and the influence cannot be ignored in the analysis of particle images. Besides the influence of the algorithm, the main reason for generating a large amount of undersampling in the particle image segmentation process is the complexity of the particle object itself, including the shape, texture, spatial position distribution and the like of the particle. The complicated texture and irregular shape greatly increase the labeling difficulty and cost of the particle image, and a large amount of labeled data is difficult to obtain. The irregular particle image segmentation can be roughly divided into three types in spatial distribution, namely particle blocking (the particle overlapping rate is low), particle overlapping but overlapping information is available, and particle overlapping and overlapping information is difficult to use. The segmentation problem for the third grain image is one of the most difficult challenges. That is, in the image, a plurality of particles are highly overlapped with each other, and block part of edge information between them, and texture information of the overlapped area cannot be obtained. Therefore, under the limited labeled data, a segmentation method for the third type of irregular particle image needs to be proposed.
The invention with application publication number CN113516130A discloses a semi-supervised image semantic segmentation method based on entropy minimization, which provides a feature gradient mapping regularization strategy (FGMR) which uses gradient mapping of low-layer feature maps in an encoder to enhance the encoding capability of the encoder on deep-layer feature maps; then, a self-adaptive sharpening strategy is provided, and the decision boundary of the unmarked data is kept in a low-density area; and in order to further reduce the influence of noise, a low-confidence consistency strategy is provided to ensure the consistency of classification and segmentation.
The patent with application publication number CN112381098A discloses a semi-supervised learning method and system based on self-learning in the field of target segmentation, and the invention trains an initial segmentation network by using data with marks in a training data set; generating pseudo labels for unlabeled data in a training data set through the trained initial segmentation network; carrying out shape quality evaluation and semantic quality evaluation on the generated pseudo label; fusing the shape quality and the semantic quality to obtain the quality of a pseudo label; estimating the distribution of the real labels and the pseudo labels, and optimizing the distribution of the pseudo labels; adding the data with higher pseudo label quality into a training data set to expand the training data set; optimizing the initial segmentation network after training by using the expanded training data set; and iterating and repeating the steps until the performance of the segmentation network is saturated. A corresponding system, terminal and medium are also provided. The patent does not specifically teach a method of generating a pseudo tag.
Disclosure of Invention
The semi-supervised hybrid training semantic segmentation method and the semi-supervised hybrid training semantic segmentation system provided by the invention solve the technical problem of low segmentation precision of the existing stacked particle images.
In order to solve the technical problem, the semi-supervised hybrid training semantic segmentation method provided by the invention comprises the following steps:
collecting the highly overlapped grain image RGB data sets, and labeling the grain image RGB data sets to obtain labeled data and non-labeled data;
establishing a pre-segmentation model based on a segmentation algorithm of an encoding and decoding frame, and training the pre-segmentation model by adopting labeled data to obtain a rough segmentation model;
identifying the label-free data by adopting a rough segmentation model to obtain a rough segmentation result, and obtaining a pseudo label based on the rough segmentation result;
training the rough segmentation model based on the pseudo labels and the labeled data to obtain a fine segmentation model;
and performing image segmentation on the highly overlapped grain images based on the fine segmentation model.
Further, labeling the grain image RGB data set, and obtaining labeled data includes:
marking the grain edges of the grain images in the grain image RGB data set;
merging the particle edges, particle gaps and other non-particle areas in the particle image to be used as a background, and using the internal area of the particles as a foreground target to obtain labeled data;
the tagged data is divided into first tagged data, second tagged data, and third tagged data.
Further, training the pre-segmentation model by adopting the data with the labels, and specifically obtaining the rough segmentation model comprises the following steps:
and training the pre-segmentation model by adopting the first labeled data to obtain a rough segmentation model.
Further, obtaining the pseudo label based on the coarse segmentation result comprises:
extracting overlapped grain connected domains from the rough segmentation result;
extracting convex hulls and convex defects of the particle connected domain, and extracting characteristic points based on the convex hulls and the convex defects;
matching the characteristic points, and obtaining segmentation line segments according to the matching result;
and overlapping the segmentation line segment and the rough segmentation result to obtain a pseudo label.
Further, extracting convex hulls and convex defects of the particle connected domain, and extracting feature points based on the convex hulls and the convex defects includes:
selecting the farthest point of the corresponding contour point in the convex defect as a characteristic point, wherein the calculation formula of the characteristic point is as follows:
Figure BDA0003490423510000031
wherein, VnRepresenting a feature point set in the nth convex hull, m representing m convex defects in the convex hull, i representing the ith convex defect, pn,iRepresenting contour points on the i-th convex defect in the n-th convex hull, CnDenotes the nth convex hull, d (p)n,i,Cn) Indicating wheelContour point pn,iTo the corresponding convex hull CnThe euclidean distance of the edge of (a).
Further, matching the feature points, and obtaining segment segments according to the matching result includes:
matching the feature points, wherein a calculation formula for matching the feature points is as follows:
Figure BDA0003490423510000032
wherein, XiA matching coefficient representing the ith feature point, wherein the matching coefficient specifically refers to the maximum matching possibility between the ith feature point and other feature points, viIndicates the ith feature point, vjRepresents the j-th feature point, i ≠ j, VnRepresenting a set of feature points in the nth convex hull, d (v)i,vj) Denotes viAnd vjEuclidean distance of, A (v)i,vj) Representing the angle formed by two line segments, one of which is defined by the characteristic point viAnd
Figure BDA0003490423510000033
and
Figure BDA0003490423510000034
is formed by the feature points vjAnd
Figure BDA0003490423510000035
and
Figure BDA0003490423510000036
is formed in the middle of the line of drawing,
Figure BDA0003490423510000037
represents the contour starting point of the i-th convex defect of the n-th convex hull,
Figure BDA0003490423510000038
represents the contour end point of the i-th convex defect of the n-th convex hull,
Figure BDA0003490423510000039
indicating the start of a contour
Figure BDA00034904235100000310
To the corresponding end point of the profile
Figure BDA00034904235100000311
Euclidean distance of, λ1,λ2The method comprises the following steps of representing an experiment parameter for adjusting the size of the whole geometric index, wherein l () represents a divided line segment formed by two points, and an angle represents an included angle formed by two line segments;
calculating the matching coefficient of each feature point, comparing the matching coefficient with a preset matching threshold, if the matching coefficient is greater than the matching threshold, connecting the two feature points in a mode of generating a two-point line segment, if the matching coefficient is less than the matching threshold, skipping the point, and completing the matching of all the feature points in a circulating mode so as to obtain the segmentation line segment.
Further, training the rough segmentation model based on the pseudo label and the labeled data, and obtaining the fine segmentation model specifically comprises:
training the rough segmentation model based on the pseudo label and the second labeled data to obtain a secondary fine segmentation model;
and training the secondary fine segmentation model based on the third labeled data to obtain a fine segmentation model.
Further, the calculation formula of the loss function for training the rough segmentation model and training the sub-fine segmentation model is as follows:
Loss(Y)＝LCE(L)+λ·LCE(P)，
where Y represents all samples, loss (Y) represents the overall loss function, L represents the tagged data set, LCE(L) represents a cross-entropy loss function trained with tagged datasets, P represents an unlabeled dataset, λ represents a training parameter, LCE(P) represents a cross-entropy loss function trained by the unlabeled dataset.
Further, obtaining the fine segmentation model further comprises:
evaluating precision of the fine segmentation model by mIOU, and evaluating degree of under-segmentation of the segmentation result by using degree of under-segmentation evaluation index
The semi-supervised hybrid training semantic segmentation system provided by the invention comprises:
the semi-supervised hybrid training semantic segmentation method comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the steps of the semi-supervised hybrid training semantic segmentation method when executing the computer program.
Compared with the prior art, the invention has the advantages that:
the invention provides a semi-supervised hybrid training semantic segmentation method and system, which acquire a highly overlapped particle image RGB data set and label the particle image RGB data set to obtain labeled data and unlabelled data, establish a pre-segmentation model based on a segmentation algorithm of an encoding and decoding frame, train the pre-segmentation model by adopting the labeled data to obtain a rough segmentation model, recognize the unlabelled data by adopting the rough segmentation model to obtain a rough segmentation result, obtain a pseudo label based on the rough segmentation result, train the rough segmentation model based on the pseudo label and the labeled data to obtain a fine segmentation model and perform image segmentation based on the fine segmentation model on the highly overlapped particle image, solve the technical problem of low segmentation precision of the stacked particle image in the prior art, fully utilize implicit information contained in the unlabelled data through the training process of the pre-segmentation model, the hybrid training and the fine tuning, and when no-label data is trained, a convex hull-based pseudo label generation algorithm is provided, the rough segmentation result obtained on the pre-segmentation model is further segmented to obtain a pseudo label which is closer to a real label in structural integrity, and finally high-precision segmentation of the stacked particle image is realized, the structural integrity of the particles can be greatly improved, and the degree of under-segmentation is reduced.
The key points of the invention comprise:
(1) a semi-supervised hybrid training framework aiming at the semantic segmentation of the complex particle images is innovatively provided, namely a semi-supervised training mode consisting of a pre-segmentation model, hybrid training and fine tuning is provided.
(2) A trainable and learnable pseudo label generation algorithm aiming at particle image segmentation is innovatively provided, a convex hull and relevant geometric characteristics on a segmented image are discovered, corresponding characteristic points are extracted and matched, and then a re-segmentation line segment is generated to obtain a pseudo label with the structural integrity closer to a real label.
(3) The semi-supervised hybrid training semantic segmentation frame and system for generating the pseudo labels based on the convex hulls are innovatively provided, efficient training of label-free data is achieved, segmentation precision close to complete supervision is achieved, structural integrity of particles can be greatly improved, and the degree of under-segmentation is reduced.
Drawings
Fig. 1 is an overall structure diagram of a semi-supervised hybrid training semantic segmentation framework for generating pseudo labels based on convex hulls in a second embodiment of the present invention;
FIG. 2 illustrates a second embodiment of the present invention, in which the coarsely partitioned convex hulls and associated geometric features are combined;
FIG. 3 is a diagram illustrating feature points and segment lines generated by matching in rough segmentation according to a second embodiment of the present invention;
fig. 4 is a graph showing comparison results of segmentation of stacked grain images using different methods.
Fig. 5 is a block diagram of a semi-supervised hybrid training semantic segmentation system according to an embodiment of the present invention.
Reference numerals:
100. a memory; 200. a processor.
Detailed Description
In order to facilitate an understanding of the invention, the invention will be described more fully and in detail below with reference to the accompanying drawings and preferred embodiments, but the scope of the invention is not limited to the specific embodiments below.
The embodiments of the invention will be described in detail below with reference to the drawings, but the invention can be implemented in many different ways as defined and covered by the claims.
Example one
The semi-supervised hybrid training semantic segmentation method provided by the embodiment of the invention comprises the following steps:
step S101, collecting a highly overlapped grain image RGB data set, and labeling the grain image RGB data set to obtain labeled data and non-labeled data;
step S102, establishing a pre-segmentation model based on a segmentation algorithm of an encoding and decoding frame, and training the pre-segmentation model by adopting labeled data to obtain a rough segmentation model;
step S103, identifying the label-free data by adopting a rough segmentation model to obtain a rough segmentation result, and obtaining a pseudo label based on the rough segmentation result;
step S104, training the rough segmentation model based on the pseudo labels and the labeled data to obtain a fine segmentation model;
and step S105, performing image segmentation on the highly overlapped grain images based on the fine segmentation model.
The semi-supervised hybrid training semantic segmentation method provided by the embodiment of the invention acquires a highly overlapped particle image RGB data set, labels the particle image RGB data set to obtain labeled data and unlabelled data, establishes a pre-segmentation model based on a segmentation algorithm of an encoding and decoding frame, trains the pre-segmentation model by adopting the labeled data to obtain a rough segmentation model, identifies the unlabelled data by adopting the rough segmentation model to obtain a rough segmentation result, obtains a pseudo label based on the rough segmentation result, trains the rough segmentation model based on the pseudo label and the labeled data to obtain a fine segmentation model and perform image segmentation on the highly overlapped particle image based on the fine segmentation model, solves the technical problem of low segmentation precision of the stacked particle image in the prior art, and fully utilizes implicit information contained in the unlabelled data through the training process of the pre-segmentation model, hybrid training and fine tuning, and when no-label data is trained, a convex hull-based pseudo label generation algorithm is provided, the rough segmentation result obtained on the pre-segmentation model is further segmented to obtain a pseudo label which is closer to a real label in structural integrity, and finally high-precision segmentation of the stacked particle image is realized, the structural integrity of the particles can be greatly improved, and the degree of under-segmentation is reduced.
Example two
The second embodiment of the invention provides a semi-supervised hybrid training semantic segmentation framework for generating pseudo labels based on convex hulls, and the algorithm of the second embodiment of the invention comprises the following steps:
(1) highly overlapping grain image RGB datasets are prepared. N RGB data are prepared for training, including 0.2n labeled samples L and 0.8n unlabeled data, denoted as U. And further dividing the 0.2n labeled data into three parts: 0.1n, 0.05n and 0.05n, respectively, are marked as L1、L2And L3Respectively applied to pre-segmentation models, hybrid training and fine tuning.
(2) And constructing a pre-segmentation model. Using tagged data L based on algorithm based on coding and decoding frame1Pre-training is carried out to obtain a pre-segmentation model f for coarse segmentationpre。
(3) A pseudo tag is generated. Roughly dividing result S of unlabeled sample U in training processcConverted to the pseudo-tag P. The generation algorithm is divided into four steps, firstly, in the course of rough segmentation result ScExtracting overlapped particle connected domain OnThen extracting its convex hull C and convex defect CdefExtracting all characteristic points V on the convex defectnThen, based on the proposed matching index X of the branch geometric features of the skeleton diagram, matching the feature points, and finally, matching the successfully matched feature points
Figure BDA0003490423510000061
Connected in the form of line segments LsAnd the result S of rough segmentationcSuperposed to form a fine division SfAnd added as a pseudo label P for subsequent training.
(4) Generating pseudo labels P and L2Random mixing as a mixed data set for further training of rough segmentation model fpreTo obtain a model fM. And adds the loss of the pseudo tag data to the overall loss function. After the mixed data set training is completed, L is used3Finally, the model is finely adjusted to obtain a final segmented model fe。
(5) And finally, evaluating the precision of the model through the mIOU, and evaluating the undersampling degree of the final segmentation result by using the undersampling degree.
The specific implementation scheme is as follows:
1. highly overlapping irregular grain image RGB datasets are prepared.
1) Selecting particle objects according to classification of particles
The particles are of different kinds and the particle objects selected by the invention are irregularly shaped, highly stacked and overlapping area information difficult to utilize.
2) Producing a data set
And acquiring an image of the particle pile through a camera, and labeling a part of samples of the particle image by using a labeling tool to be used as labeled samples in the data set, and using the rest samples as unlabeled samples.
3) Partitioning a data set
If there are n RGB data, 0.2n of the RGB data is used as a labeled sample L, and 0.8n of the RGB data is used as unlabeled data and is marked as U. Meanwhile, 0.2n labeled data is further divided into three parts: 0.1n, 0.05n and 0.05n, respectively, are marked as L1、L2And L3。
2. Pre-segmentation model f for constructing segmentation algorithm based on coding and decoding frameworkpre。
On the basis of an encoding and decoding frame, the invention builds an image segmentation pre-segmentation model f with a four-layer structurepreThe overall architecture of the network is shown in fig. 1. The network consists of two parts: an encoder and a decoder.
1) Encoder for encoding a video signal
In the encoder, each layer is composed of a pooling feature extraction module consisting of two groups of convolution of 3 × 3, BN (batch normalization) and a ReLU activation function, and the two layers are subjected to dimension reduction by using the maximum pooling. Inputting an RGB image of 3 s, and outputting an RGB feature tensor U of s (s/16) through an encoderrgb＝(u0,u1,...,us-1)。
2) Decoder
The decoding process adoptsSimilar to the left compression branch, each layer has upsampling, two sets of convolutions, BN and ReLU activation functions, for a four-layer structure. Frgb＝(u0,0,u1,0,...,us,0) Inputting the data into a decoder, performing four times of pooling dimension reduction (maximum pooling) and 3 x 3 convolution operations, and finally outputting a binary rough segmentation result S with the size of 1 x Sc。
3) Classification target
In general, the classification target of image segmentation is to use the edge of a particle as a foreground and the other regions as a background. This segmentation currently does not distinguish whether a pixel belongs to an intra-particle region or to a gap between particles. The invention makes the edge e of the particle and the gap g of the particle and other non-particle areas anpCombined, taken as background B, and then the inner region a of the particleinAs the foreground target T, the edges of the particles need to be marked separately in the marking process. This embodiment is through adopting this kind of classification mode, has avoided present edge as the prospect with the granule, and the classification mode that other regions are as the background leads to can't distinguish that the pixel belongs to granule inner zone or belongs to the gap between the granule.
3. Convex hull-based pseudo label generation algorithm
The label-free data U is converted into the pseudo label P in the training process through a re-segmentation method, the purpose is to further segment the image by using the geometric characteristics of the roughly segmented image under the limited labels, so that the image is formed into the pseudo label with the structural integrity closer to a real label, the further training of the model is facilitated, the effect close to complete supervision is achieved, and the occurrence of under-segmentation is reduced. The generation algorithm is divided into four steps.
1) Identifying overlapping particle connected domains On
In order to increase the calculation speed of the algorithm, each particle connected domain does not need to be calculated, and only the particle connected domain O which is overlapped needs to be researchedn. And performing cluster analysis on all particle connected domains by using a K-means clustering algorithm.
Step 1: selecting and initializing 2 samples as initial clustering center a ═ a1,a2；
Step 2: for each sample x in the datasetiCalculating the distance from the cluster center to the 2 cluster centers and dividing the cluster center into the classes corresponding to the cluster centers with the minimum distance;
step 3: for each class ajRecalculating its cluster center
Figure BDA0003490423510000081
(i.e., the centroids of all samples belonging to the class);
step 4: the above 2, 3 steps are repeated until a certain termination condition (number of iterations, minimum error variation, etc.) is reached.
Extracting overlapped particle connected domain O by clusteringn。
2) Extracting characteristic point set V on rough segmentation imagen
Firstly, the result S of rough segmentationcCalculating to obtain each overlapped particle connected domain OnConvex hull C ofnAnd all convex defects D thereofn,mThen, corresponding contour points p in each convex defect are setn,mAs a feature point in an overlapping particle connected component domain, as shown in fig. 2, fig. 2 shows a roughly segmented convex hull and associated geometric features.
Figure BDA0003490423510000082
Wherein d (p)n,i,Cn) Representing a contour point pn,iTo the corresponding convex hull CnThe euclidean distance of the edge of (a).
3) Feature point matching
Aiming at the geometric characteristics of convex hulls and convex defects in the roughly-segmented image, the invention provides a geometric index for pairwise matching of all extracted feature points, which is defined as follows:
Figure BDA0003490423510000091
wherein, XiA matching coefficient representing the ith feature point, wherein the matching coefficient specifically refers to the maximum matching possibility between the ith feature point and other feature points, viIndicates the ith feature point, vjRepresents the j-th characteristic point, i ≠ j, VnRepresenting a set of feature points in the nth convex hull, d (v)i,vj) Denotes viAnd vjEuclidean distance of, A (v)i,vj) Representing the angle formed by two line segments, one of which is defined by the characteristic point viAnd
Figure BDA0003490423510000092
and
Figure BDA0003490423510000093
is formed by the feature points vjAnd
Figure BDA0003490423510000094
and
Figure BDA0003490423510000095
is formed in the middle of the line of drawing,
Figure BDA0003490423510000096
represents the contour starting point of the i-th convex defect of the n-th convex hull,
Figure BDA0003490423510000097
represents the contour end point of the i-th convex defect of the n-th convex hull,
Figure BDA0003490423510000098
indicating the start of a contour
Figure BDA0003490423510000099
To the corresponding end point of the profile
Figure BDA00034904235100000910
Euclidean distance of, λ1,λ2An experimental parameter representing the magnitude of the overall geometric index to be adjusted, l () represents a divided line segment formed by two points, and angle represents a line segment formed by two line segmentsThe included angle is formed;
in a real vector space V, for a given set X, the intersection S of all convex sets containing X is called the convex hull of X. Any deviation of a contour from its convex hull is referred to as a convex defect.
The reason for selecting the convex hull and other methods in this embodiment is mainly based on the following points:
1. in order to solve the problem of overlapped particle segmentation, the segmentation precision needs to be improved, and the degree of under-segmentation needs to be reduced, wherein the simplest method is to generate a re-segmentation line segment;
2. the generation of the subdivision line segments requires the determination of starting and ending points, and only depends on the geometric features on the graph, which are provided by convex hulls and convex defects;
3. other pseudo label methods are more general methods driven by data, and are not suitable for the technical problems provided by the embodiment of the invention, and the method for generating the pseudo label based on the convex hull adopted by the embodiment can more quickly, simply and accurately find the required characteristics, thereby being beneficial to improving the model training precision and further improving the segmentation precision of the stacked particle images.
4) Generating a segment Ls
Calculating X of each characteristic pointiAnd comparing the characteristic points with a matching threshold alpha, if the characteristic points are larger than or equal to the value, connecting the two characteristic points in a mode of generating line segments of the two points, and if the characteristic points are smaller than the value, skipping the points. Finally, all the segment segments L are completed by loopingsAs shown in fig. 3, fig. 3 shows the feature points and the matching generated segment line in the rough segmentation. And mixing LsAnd the result S of rough segmentationcAnd performing superposition to form a final pseudo label P.
P＝fpre(U)+Ls＝Sc+Ls，
The pseudo label generation algorithm is used for obtaining the pseudo label which has higher precision and is closer to a real label in structural integrity on the basis of rough segmentation.
4. Semi-supervised hybrid training framework
First, if there are n RGB data, it will beWherein 0.2n samples are labeled as labeled samples L, and 0.8n samples are unlabeled and marked as U. Meanwhile, 0.2n labeled data is further divided into three parts: 0.1n, 0.05n and 0.05n, respectively, are marked as L1、L2And L3。
fpre＝f(L1)，
Secondly, mixing L2And the non-label data U are randomly and uniformly mixed to form a mixed data set M, and then the mixed data set M is subjected to fpreAnd continuing training on the basis, if no label data is trained, calling a pseudo label generation algorithm to generate a pseudo label, and using the pseudo label as a real label. After training the complete mixed dataset, model f is obtainedM。
fM＝fpre(M)，
Finally, the remaining tagged data L is used3In model fMFine tuning is carried out to obtain a final high-precision segmentation model fe。
fe＝fM(L3)，
The loss of pseudo-label data is added to the overall loss function throughout the hybrid training process. The global loss function is defined as:
Loss(Y)＝LCE(L)+λ·LCE(P)，
where Y represents all samples, loss (Y) represents the overall loss function, L represents the tagged data set, LCE(L) represents a cross-entropy loss function trained with tagged datasets, P represents an unlabeled dataset, λ represents a training parameter, LCE(P) represents a cross-entropy loss function trained by the unlabeled dataset.
The invention provides a semi-supervised hybrid training semantic segmentation frame and system for generating pseudo labels based on convex hulls, which are used for improving the precision of segmenting particle images and reducing under-segmentation and unseparated connected domains in the segmentation process.
The invention provides a semi-supervised hybrid training semantic segmentation framework and a system for generating pseudo labels based on convex hulls. Compared with the existing method, the invention innovatively provides a semi-supervised hybrid training framework aiming at complex particle image segmentation, and the hidden information contained in the label-free data is fully utilized through the training processes of pre-segmentation model, hybrid training and fine tuning. And when the label-free data is trained, a convex hull-based pseudo label generation algorithm is provided, the rough segmentation result obtained on the pre-segmentation model is further segmented, and a pseudo label which is closer to a real label in structural integrity is obtained. Finally, the segmentation precision close to complete supervision is realized, the structural integrity of the particles can be greatly improved, and the degree of under-segmentation is reduced.
Example 3
This example uses 2650m in a certain iron works3And verifying the material image of the large-scale blast furnace. The invention provides a semi-supervised hybrid training semantic segmentation framework and a system for generating pseudo labels based on convex hulls, which specifically comprise the following steps:
1. and (5) making a data set. In order to obtain RGB pictures of blast furnace materials, RGB image information is collected on a blast furnace feeding belt of a certain iron and steel plant through hardware equipment such as an industrial camera. The industrial camera selects the data acquisition software of the German Basler acA2500-14gm Basler industrial area-array camera as pylon Viewer. And the communication is carried out in an Ethernet mode. A total of 8 sets of data (one set of data was collected each time, one ore batch was replaced) were collected by an industrial camera in a centralized time period, each set of data comprising 1000 and 2000 pictures, wherein the size of the pictures was 2592 x 2048.
2. Semi-supervised hybrid training to generate pseudo labels based on convex hulls. First, if 8000 RGB data are currently owned, 1600 of the RGB data are labeled as labeled samples L, and 6400 samples are labeled as unlabeled data, which is denoted as U. Meanwhile, 1600 tagged data are further divided into three parts: 800. 400 and 400, respectively denoted as L1、L2And L3. Then theAnd designing coding branches in a pre-segmentation model, wherein each layer consists of a feature extraction module consisting of 2 x 2 pooling, two groups of 3 x 3 convolution, BN (batch normalization) and a ReLU activation function, and the maximum pooling is used between the two layers for dimension reduction. The right decoding process employs a similar strategy as the left compression branch, with each layer having an upsampling, two sets of convolutions, BN and ReLU activation functions. Training was performed with 400 tape-labeled data. In the training process, Adam is adopted as an optimization algorithm. The learning rate is set to 10^ -4, wherein the learning rate is adjusted by adopting a cosine annealing learning rate scheduling strategy, and the minimum value is 10^ -6. The weight attenuation is set to 10-8. A pre-segmentation model capable of carrying out two classifications is obtained through training, and the purpose is to divide pixels in a blast furnace material image into a background and material particles. Next, 400 tagged data and 6400 untagged data are randomly and uniformly mixed to form a mixed data set M with a size of 6800, and then at fpreAnd continuing training on the basis, if no label data is trained, calling a pseudo label generation algorithm to generate a pseudo label, and using the pseudo label as a real label. After training the complete mixed dataset, model f is obtainedM. Finally, the remaining 400 tagged data L is used3In model fMFine tuning is carried out to obtain a final high-precision segmentation model fe。
3. Experimental results and analysis. On the same experimental platform, the labeled data set was divided into three parts according to different proportions, and comparative experiments were performed using U-Net, MaskRCNN, ALSS and the methods herein. Fig. 4 shows comparison results in different methods, where (a) in fig. 4 shows an original image, (b) a segmentation result map by the UNet method, (c) a segmentation result map by the maskrnn method, (d) a segmentation result map by the method according to the embodiment of the present invention, (e) a skeleton map by the method according to the embodiment of the present invention, and (f) a true label map. Table 1 shows the performance of the common evaluation index of image segmentation under different methods and different labeled data ratios. As can be known from the segmentation indexes in the table 1, the segmentation results of the semi-supervised hybrid training semantic segmentation framework and the system for generating the pseudo labels based on the convex hulls are superior to those of other deep learning models. Meanwhile, the method of the invention carries out comparison experiments under different proportions of 4:0:0, 2:2:0, 2:0:2 and 2:1:1, and finally obtains that 2:1:1 is the current optimal division proportion.
TABLE 1
Figure BDA0003490423510000121
Referring to fig. 5, a semi-supervised hybrid training semantic segmentation system provided by the embodiment of the present invention includes:
a memory 10, a processor 20, and a computer program stored on the memory 10 and executable on the processor 20, wherein the processor 20 implements the steps of the semi-supervised hybrid training semantic segmentation method proposed by the present embodiment when executing the computer program.
The specific working process and working principle of the semi-supervised hybrid training semantic segmentation system in this embodiment may refer to the working process and working principle of the semi-supervised hybrid training semantic segmentation method in this embodiment.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A semi-supervised hybrid training semantic segmentation method, characterized by comprising:
collecting the highly overlapped grain image RGB data sets, and labeling the grain image RGB data sets to obtain labeled data and non-labeled data;
establishing a pre-segmentation model based on a segmentation algorithm of an encoding and decoding frame, and training the pre-segmentation model by adopting labeled data to obtain a rough segmentation model;
identifying label-free data by adopting a rough segmentation model to obtain a rough segmentation result, and obtaining a pseudo label based on the rough segmentation result;
training the rough segmentation model based on the pseudo labels and the labeled data to obtain a fine segmentation model;
and performing image segmentation on the highly overlapped grain images based on the fine segmentation model.
2. The semi-supervised hybrid training semantic segmentation method of claim 1, wherein labeling a grain image RGB data set to obtain labeled data comprises:
marking the grain edges of the grain images in the grain image RGB data set;
merging the particle edges, particle gaps and other non-particle areas in the particle image to be used as a background, and using the internal area of the particles as a foreground target to obtain labeled data;
the tape-flag data is divided into first tape-flag data, second tape-flag data, and third tape-flag data.
3. The semi-supervised hybrid training semantic segmentation method according to claim 2, wherein the pre-segmentation model is trained by using labeled data, and the obtained rough segmentation model specifically comprises:
and training the pre-segmentation model by adopting the first labeled data to obtain a rough segmentation model.
4. The semi-supervised hybrid training semantic segmentation method of any one of claims 1-3, wherein obtaining pseudo labels based on the coarse segmentation result comprises:
extracting overlapped grain connected domains from the rough segmentation result;
extracting convex hulls and convex defects of the particle connected domain, and extracting characteristic points based on the convex hulls and the convex defects;
matching the characteristic points, and obtaining segmentation line segments according to the matching result;
and overlapping the segmentation line segment and the rough segmentation result to obtain a pseudo label.
5. The semi-supervised hybrid training semantic segmentation method according to claim 4, wherein extracting convex hulls and convex defects of a particle connected component, and extracting feature points based on the convex hulls and the convex defects comprises:
selecting the farthest point of the corresponding contour point in the convex defect as a characteristic point, wherein the calculation formula of the characteristic point is as follows:
Figure FDA0003490423500000011
wherein, VnRepresenting the feature point set in the nth convex hull, m representing m convex defects in the convex hull, i representing the ith convex defect, pn,iRepresenting contour points on the i-th convex defect in the n-th convex hull, CnDenotes the nth convex hull, d (p)n,i,Cn) Representing a contour point pn,iTo the corresponding convex hull CnThe euclidean distance of the edge of (a).
6. The semi-supervised hybrid training semantic segmentation method of claim 5, wherein matching feature points and obtaining segment segments according to matching results comprises:
matching the feature points, wherein a calculation formula for matching the feature points is as follows:
Figure FDA0003490423500000021
wherein, XiA matching coefficient representing the ith feature point, wherein the matching coefficient specifically refers to the maximum matching possibility between the ith feature point and other feature points, viIndicates the ith feature point, vjRepresents the j-th characteristic point, i ≠ j, VnRepresenting a set of feature points in the nth convex hull, d (v)i,vj) Denotes viAnd vjEuclidean distance of, A (v)i,vj) Representing the angle formed by two line segments, one of which is defined by the characteristic point viAnd
Figure FDA0003490423500000022
and
Figure FDA0003490423500000023
is formed by the feature points vjAnd
Figure FDA0003490423500000024
and
Figure FDA0003490423500000025
is formed in the middle of the line of drawing,
Figure FDA0003490423500000026
represents the contour starting point of the i-th convex defect of the n-th convex hull,
Figure FDA0003490423500000027
represents the contour end point of the i-th convex defect of the n-th convex hull,
Figure FDA0003490423500000028
indicating the start of a contour
Figure FDA0003490423500000029
To the corresponding end point of the profile
Figure FDA00034904235000000210
Euclidean distance of, λ1,λ2The method comprises the following steps of representing an experiment parameter for adjusting the size of the whole geometric index, wherein l () represents a divided line segment formed by two points, and an angle represents an included angle formed by two line segments;
and calculating a matching coefficient of each feature point, comparing the matching coefficient with a preset matching threshold, if the matching coefficient is greater than the matching threshold, connecting the two feature points in a manner of generating a two-point line segment, if the matching coefficient is less than the matching threshold, skipping the point, and completing matching of all the feature points in a circulating manner to obtain a segmentation line segment.
7. The semi-supervised hybrid training semantic segmentation method according to claim 6, wherein the coarse segmentation model is trained based on pseudo labels and labeled data, and the obtaining of the fine segmentation model specifically comprises:
training the rough segmentation model based on the pseudo label and the second labeled data to obtain a secondary fine segmentation model;
and training the secondary fine segmentation model based on the third labeled data to obtain a fine segmentation model.
8. The semi-supervised hybrid training semantic segmentation method according to claim 7, wherein the computational formula of the loss function for training the rough segmentation model and training the sub-fine segmentation model is as follows:
Loss(Y)＝LCE(L)+λ·LCE(P)，
where Y represents all samples, loss (Y) represents the overall loss function, L represents the tagged data set, LCE(L) represents a cross-entropy loss function trained with tagged datasets, P represents an unlabeled dataset, λ represents a training parameter, LCE(P) represents a cross-entropy loss function trained by the unlabeled dataset.
9. The semi-supervised hybrid training semantic segmentation method according to claim 8, wherein obtaining the fine segmentation model further comprises:
the precision of the fine segmentation model is evaluated by the mIOU, and the degree of under-segmentation of the segmentation result is evaluated by the under-segmentation degree evaluation index.
10. A semi-supervised hybrid training semantic segmentation system, the system comprising:
memory (100), processor (200) and a computer program stored on the memory (100) and executable on the processor (200), characterized in that the processor (200) implements the steps of the method of any of the preceding claims 1 to 9 when executing the computer program.
CN202210094172.4A 2022-01-26 2022-01-26 Semi-supervised hybrid training semantic segmentation method and system Pending CN114549834A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210094172.4A CN114549834A (en) 2022-01-26 2022-01-26 Semi-supervised hybrid training semantic segmentation method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210094172.4A CN114549834A (en) 2022-01-26 2022-01-26 Semi-supervised hybrid training semantic segmentation method and system

Publications (1)

Publication Number Publication Date
CN114549834A true CN114549834A (en) 2022-05-27

Family

ID=81674493

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210094172.4A Pending CN114549834A (en) 2022-01-26 2022-01-26 Semi-supervised hybrid training semantic segmentation method and system

Country Status (1)

Country Link
CN (1) CN114549834A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115147426A (en) * 2022-09-06 2022-10-04 北京大学 Model training and image segmentation method and system based on semi-supervised learning
CN115272311A (en) * 2022-09-26 2022-11-01 江苏亚振钻石有限公司 Wolframite image segmentation method based on machine vision
CN115900586A (en) * 2022-11-24 2023-04-04 合肥工业大学 Reclaimed sand particle morphology real-time monitoring device and real-time monitoring method

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115147426A (en) * 2022-09-06 2022-10-04 北京大学 Model training and image segmentation method and system based on semi-supervised learning
CN115147426B (en) * 2022-09-06 2022-11-29 北京大学 Model training and image segmentation method and system based on semi-supervised learning
CN115272311A (en) * 2022-09-26 2022-11-01 江苏亚振钻石有限公司 Wolframite image segmentation method based on machine vision
CN115900586A (en) * 2022-11-24 2023-04-04 合肥工业大学 Reclaimed sand particle morphology real-time monitoring device and real-time monitoring method

Similar Documents

Publication Publication Date Title
Adarsh et al. YOLO v3-Tiny: Object Detection and Recognition using one stage improved model
CN109190752B (en) Image semantic segmentation method based on global features and local features of deep learning
CN114549834A (en) Semi-supervised hybrid training semantic segmentation method and system
CN114120102A (en) Boundary-optimized remote sensing image semantic segmentation method, device, equipment and medium
CN111311563A (en) Image tampering detection method based on multi-domain feature fusion
CN112381788B (en) Part surface defect increment detection method based on double-branch matching network
Zheng et al. HLU 2-Net: a residual U-structure embedded U-Net with hybrid loss for tire defect inspection
CN116258719B (en) Flotation foam image segmentation method and device based on multi-mode data fusion
CN113160139B (en) Attention-based steel plate surface defect detection method of Faster R-CNN network
CN112200186B (en) Vehicle logo identification method based on improved YOLO_V3 model
CN105005794A (en) Image pixel semantic annotation method with combination of multi-granularity context information
Liu et al. A classification method of glass defect based on multiresolution and information fusion
CN110008899B (en) Method for extracting and classifying candidate targets of visible light remote sensing image
CN112766170B (en) Self-adaptive segmentation detection method and device based on cluster unmanned aerial vehicle image
CN104268600A (en) Mineral flotation froth image texture analysis and working condition identification method based on Minkowski distance
CN113159215A (en) Small target detection and identification method based on fast Rcnn
CN111161213A (en) Industrial product defect image classification method based on knowledge graph
CN113971735A (en) Depth image clustering method, system, device, medium and terminal
CN113657414B (en) Object identification method
CN113408651B (en) Unsupervised three-dimensional object classification method based on local discriminant enhancement
Ying et al. License plate detection and localization in complex scenes based on deep learning
Peng et al. Semi-supervised bolt anomaly detection based on local feature reconstruction
CN111612076B (en) Image fine recognition method based on DCL and Cascade
CN112070116A (en) Automatic art painting classification system and method based on support vector machine
CN111832463A (en) Deep learning-based traffic sign detection method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination