CN114066867A - Deep learning-based crack propagation trace missing region segmentation method - Google Patents

Deep learning-based crack propagation trace missing region segmentation method Download PDF

Info

Publication number
CN114066867A
CN114066867A CN202111392850.7A CN202111392850A CN114066867A CN 114066867 A CN114066867 A CN 114066867A CN 202111392850 A CN202111392850 A CN 202111392850A CN 114066867 A CN114066867 A CN 114066867A
Authority
CN
China
Prior art keywords
missing
crack propagation
image
data set
missing region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111392850.7A
Other languages
Chinese (zh)
Inventor
张啸尘
李翰文
张天
孟维迎
龙彦泽
李颂华
周鹏
石怀涛
丁兆洋
张宇
邹德芳
李峻州
范才子
金兰茹
刁梦楠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenyang Jianzhu University
Original Assignee
Shenyang Jianzhu University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenyang Jianzhu University filed Critical Shenyang Jianzhu University
Priority to CN202111392850.7A priority Critical patent/CN114066867A/en
Publication of CN114066867A publication Critical patent/CN114066867A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G06T7/0008—Industrial image inspection checking presence/absence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20092—Interactive image processing based on input by user
    • G06T2207/20104—Interactive definition of region of interest [ROI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a deep learning-based crack propagation trace missing region segmentation method, and relates to the technical field of computer vision. Firstly, preprocessing an image containing a crack propagation trace missing area, marking an outer frame of a part of a crack propagation trace missing area data set image by using a marking tool, inputting the part of the crack propagation trace missing area data set image into an auxiliary segmentation model for weak supervision training, and inputting a mask semantic segmentation image into a missing area semantic segmentation model; the other part of crack propagation trace missing area data set image is subjected to mask marking by using a marking tool and then is input into a missing area semantic segmentation model for full supervision training; fusing two parts of crack propagation trace missing region data set images input into a missing region semantic segmentation model for training to obtain a missing segmentation region; and finally, a fine contour segmentation model of the crack propagation trace missing region is formed by utilizing the generated countermeasure network, and the crack propagation trace missing region of the input picture is segmented.

Description

Deep learning-based crack propagation trace missing region segmentation method
Technical Field
The invention relates to the technical field of computer vision, in particular to a deep learning-based method for segmenting a crack propagation trace missing area.
Background
The operation reliability and the service safety of mechanical equipment directly influence national defense construction and national economic development. The large-size structure usually contains some inevitable initial defects, such as metallurgical defects, manufacturing defects and the like, and cracks are also generated under the action of random cyclic loads and complex environments in the service process. As the service life increases, these defects or cracks continue to propagate until failure at break is initiated. From a microscopic scale point of view, the study of cracks is both the most dominant "method" of failure at fracture and the only "material evidence". The method is an important means for researching the failure history and failure mechanism of the cracks by deeply analyzing the cracks, and has important guiding significance for optimizing material design and modification, structural design and manufacturing process.
Complex environmental factors, human factors and the like can damage the crack appearance to a certain degree, so that partial trace of the crack propagation process is lost. On the other hand, the texture features of the cracks are fine, complicated and irregular, and the microscopic imaging technology is difficult to present a full picture of fine textures. Such discontinuity and incompleteness of crack propagation information results in failure causes and failure mechanisms not being accurately ascertained. The method adopts a new theory and a new method to trace the missing information in the micro-scale crack propagation process, thereby recovering the failure process and identifying the failure mechanism, and is a new idea, a new concept and a new technology which have important value and bright prospect in the field of failure analysis.
However, the conventional method for segmenting the crack propagation trace missing region is manually performed, such as by using a high-resolution electron microscope and an X-ray, which requires a specialized technician and complicated equipment, and is inefficient. The detection precision facing the complex crack missing region is low, the generated missing region contour line is thick, the line edge is irregular, the contour is not smooth, the filling content is not accurate, and therefore the texture repair of the high-frequency region is incomplete. With the development of artificial intelligence and computer vision, deep learning methods are promising for solving such problems. However, the contour of the crack propagation missing region of the micro-scale fracture image is complex and changeable, the existing method usually needs a large number of finely labeled images for training, and the labeling process is tedious and consumes manpower. Therefore, how to solve the generalization problem in the process of generating the outline of the missing region, establish a depth inference network capable of accurately expressing the outline of the crack propagation missing region, judge the generation precision of the missing region and optimize the training strategy of the network is a key for realizing the accurate expression of the outline of the crack propagation missing region and is also a core task for realizing the intelligent tracing of the micro-scale crack propagation process.
Disclosure of Invention
The invention aims to solve the technical problem of providing a method for segmenting the crack propagation trace missing region based on deep learning, aiming at the defects of the prior art, and accurately extracting the crack propagation trace missing region by utilizing a deep learning algorithm frame.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows: a segmentation method of crack propagation trace missing regions based on deep learning comprises the steps of preprocessing an image containing a crack propagation trace missing region, firstly, utilizing a marking tool to mark a bounding box image of a part of a crack propagation trace missing region data set image, then inputting the image into an auxiliary segmentation model to perform weak supervision training to obtain an accurate mask semantic segmentation image, and inputting the mask semantic segmentation image into a missing region semantic segmentation model; simultaneously, the other part of the crack propagation trace missing area data set image is subjected to mask marking by using a marking tool and then is input into a missing area semantic segmentation model for full supervision training; fusing a part of crack propagation trace missing region data set images input into the missing region semantic segmentation model and trained with another part of crack propagation trace missing region data set images by using a multi-scale feature combination method to obtain missing segmentation regions; finally, a game mechanism for generating a countermeasure network is utilized to form a fine contour segmentation model of the crack propagation trace missing region, and the crack propagation trace missing region of the input picture is accurately segmented;
the method specifically comprises the following steps:
step 1: acquiring an image containing a crack propagation trace missing area to form a data set; the data set is divided into two parts: one part of the crack propagation trace missing region data set accounts for more than seventy percent of the total data set and is used for marking the outer frame bounding box, and the data set is called as a most part of crack propagation trace missing region data set; the other part is other crack propagation trace missing region data sets marked by semantic segmentation masks in the total data set, and is called a small part crack propagation trace missing region data set;
step 2: carrying out a pre-training process: carrying out outer frame bounding box marking on most crack propagation trace missing area data sets by using a marking tool, taking the gray level images of the missing areas marked by the outer frame bounding box, the marked label images and the original images containing the crack propagation trace missing areas as weak supervision training data sets, carrying out weak supervision training on the auxiliary segmentation model until the model converges, and obtaining mask label images of most crack propagation trace missing area data sets; wherein the auxiliary segmentation model employs an existing encoder-decoder segmentation model;
step 2.1: marking the outer frame bounding box of the image in the data set of the most crack propagation trace missing area by using a marking tool to obtain a texture gray image and a label image of the missing area marked by the outer frame bounding box;
step 2.2, taking an original image in a data set of a large part of crack expansion trace missing area, a texture gray image of the missing area marked by an outer frame bounding box and a label image as a weak supervision data set;
step 2.3: selecting a group of weakly supervised training data set samples from the weakly supervised data set, inputting the samples into an auxiliary segmentation model, and carrying out weakly supervised training on the auxiliary segmentation model;
step 2.4: extracting background features of the bounding box label image and image features of the missing region through a segmentation encoder in the auxiliary segmentation model, and learning a logical relation between the background features of the bounding box and the image features of the missing region to obtain a predicted missing region;
step 2.5: calculating the error between the predicted missing region and the background morphology by using a segmentation decoder, adding a self-correction module into the segmentation decoder, and distinguishing and gradually reducing the error between the predicted missing region and the background morphology by using the cross entropy loss in the self-correction module;
step 2.6: repeating the step 2.1-2.5 until the auxiliary segmentation model converges to obtain a mask label image of the data set of the mostly crack propagation trace missing area;
and step 3: carrying out a training process: carrying out pixel-level mask marking on a small part of crack propagation trace missing area data set by using a marking tool to obtain a marked missing area gray image and a mask label image; taking the marked gray level image of the missing area, the mask label image and the original image of the crack expansion trace missing area as a full-supervision training data set; inputting two types of texture gray images and mask label images of the missing region in a data set of the missing region of most crack extension traces and a data set of the missing region of small crack extension traces into a semantic segmentation model of the missing region for semi-supervised training until the model converges to obtain accurate prediction missing regions of the data set of the missing region of most crack extension traces and the data set of the missing region of small crack extension traces; wherein, a FasterR-CNN model is adopted in the semantic segmentation model of the deletion region;
the steps of training the semantic segmentation model of the missing region through the data set of the missing region of the small crack propagation trace are as follows:
step 3.1: carrying out pixel-level mask marking on the image in the data set of the small crack propagation trace missing area by using a marking tool Labelme to obtain a marked missing area gray image and a mask label image;
step 3.2: taking an original image of the crack propagation trace missing area, a gray image of the crack propagation trace missing area marked by a marking tool Labelme and a mask label image as a fully supervised training dataset;
step 3.3: selecting a group of fully supervised training data set samples from a fully supervised training data set, inputting the samples into a missing region semantic segmentation model, and training the missing region semantic segmentation model;
step 3.4: extracting background morphology and image characteristics of the missing region through a Resnet50 network in the semantic segmentation model of the missing region to obtain a shared characteristic layer; acquiring a suggestion frame by using the shared characteristic layer, decoding the suggestion frame, and intercepting the position with the missing region at the shared characteristic layer to obtain a predicted missing region;
step 3.5: calculating the coincidence degree of all the suggestion frames and the real frames by using an ROI module in the semantic segmentation model of the missing region, screening, finally adjusting and convolving, and adjusting the suggestion frames into the real frames to obtain an accurate prediction missing region;
step 3.6: repeating the steps of 3.1-3.5 until the training of the data set of the small crack propagation trace missing region is finished, and converging the semantic segmentation model of the missing region;
the steps of training the missing region segmentation model through the data set of the missing region of most crack propagation traces are as follows:
step 3.7: inputting the mask label image and the original image of the crack expansion trace missing area, which are obtained by the missing area image marked by the bounding box in the step 2 through the auxiliary segmentation model, into the missing area semantic segmentation model;
step 3.8: the rest steps are the same as the steps 3.3-3.4, until the training of most crack propagation trace missing area data sets is finished, the semantic segmentation model of the missing area is converged;
and 4, step 4: fusing most crack propagation trace missing region data set images and small crack propagation trace missing region data set images which are input and trained by a missing region semantic segmentation model by using a multi-scale feature combination method, and training a fine contour model based on a generated countermeasure network; wherein, the fine contour model based on the generation countermeasure network adopts GAN generation countermeasure network;
step 4.1: taking the original image of the crack expansion trace missing area in the step 2 and the step 3, and the gray level image and the Mask label image which are obtained respectively as training data sets, and training a fine contour model based on the generated countermeasure network;
step 4.2: selecting a group of training data sets in the step 4.1 as samples, and inputting the samples into a fine contour model based on the generated countermeasure network;
step 4.3: extracting the contour features of the background feature and the missing region based on a generator in a fine contour model of a generated countermeasure network, and learning the logical relationship between the background feature and the contour features of the missing region to obtain a predicted missing contour;
step 4.4: calculating the error of the predicted missing contour and the real original contour by using a discriminator in a fine contour model for generating the countermeasure network, distinguishing the predicted missing contour and the real original contour, and optimizing a generator;
step 4.5: repeating the steps 4.3-4.4 until the fine contour model based on the generated countermeasure network converges;
and 5: and (3) carrying out a segmentation acquisition process: taking the original image with the missing crack propagation trace, the gray image with the missing crack propagation trace and the Mask label image as input images, and performing alternate iteration and optimization on the generator and the discriminator by using the fine contour model trained in the step 4 and based on the generation countermeasure network to finally obtain a fine segmentation image of the missing area;
step 5.1: carrying out bounding box marking on the image containing the crack propagation trace missing area according to the step 2 to obtain a texture gray image and a label image of the missing area;
step 5.2: performing feature extraction on the texture gray level image and the label image of the missing region in the step 5.1 by using the auxiliary segmentation model trained in the step 2 to obtain an accurate mask label image;
step 5.3: inputting an original image containing a crack expansion trace missing area, a texture gray level image of the missing area and a mask label image into a fine contour model based on a generated countermeasure network;
step 5.4: extracting multi-scale background morphology and missing region contour characteristics based on a generator in a fine contour model for generating a countermeasure network, predicting a missing contour, and realizing reconstruction and restoration of the contour in the missing region;
step 5.5: calculating errors of the predicted missing contour and the real original contour by using a discriminator in a fine contour model based on a generated confrontation network, performing confrontation loss training, reducing the errors of the missing contour and the real contour, and optimizing a generator;
step 5.6: and finally obtaining a complete refined crack propagation characteristic segmentation contour by combining the characteristic contour of the original image and the repaired characteristic contour in the missing area.
Adopt the produced beneficial effect of above-mentioned technical scheme to lie in: according to the deep learning-based crack propagation trace missing region segmentation method, the composite edge texture of the multi-source image data training set is adjusted by inputting the incomplete texture image of the crack propagation trace missing region. And the intelligent segmentation of the crack propagation trace missing area is completed by deeply learning the characteristics of most crack propagation trace missing areas and the characteristics of a small crack propagation trace missing area, supervising the learning and generating an antagonistic game training mechanism. Compared with the traditional method, the segmentation method improves the segmentation precision, can segment the missing area in the crack propagation process, and greatly reduces the degree of manual participation.
Drawings
Fig. 1 is a flowchart of a method for segmenting a crack propagation trace missing region based on deep learning according to an embodiment of the present invention;
FIG. 2 is a process diagram of a segmentation method for crack propagation trace missing regions based on deep learning according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a pre-training process provided by an embodiment of the present invention;
fig. 4 is a schematic diagram of a multi-source fusion mechanism adopted by the generator according to the embodiment of the present invention.
Detailed Description
The following detailed description of embodiments of the present invention is provided in connection with the accompanying drawings and examples. The following examples are intended to illustrate the invention but are not intended to limit the scope of the invention.
In the embodiment, a method for segmenting a crack propagation trace missing region based on deep learning comprises the steps of preprocessing an image containing the crack propagation trace missing region, marking a part of a data set image of the crack propagation trace missing region by using a marking tool, inputting the part of the data set image into an auxiliary segmentation model for weak supervision training to obtain an accurate mask semantic segmentation image, and inputting the mask semantic segmentation image into the missing region semantic segmentation model; simultaneously, the other part of the crack propagation trace missing area data set image is subjected to mask marking by using a marking tool and then is input into a missing area semantic segmentation model for full supervision training; fusing a part of crack propagation trace missing region data set images input into the missing region semantic segmentation model and trained with another part of crack propagation trace missing region data set images by using a multi-scale feature combination method to obtain missing segmentation regions; finally, a game mechanism for generating a countermeasure network is utilized to form a fine contour segmentation model of the crack propagation trace missing region, and the crack propagation trace missing region of the input picture is accurately segmented; as shown in fig. 1 and 2, the method specifically comprises the following steps:
step 1: acquiring an image containing a crack propagation trace missing area to form a data set; the data set is divided into two parts: one part of the large crack propagation trace missing area data set accounts for more than seventy percent of the total data set and is used for marking the outer frame bounding box, and the large crack propagation trace missing area data set is called as a large part of crack propagation trace missing area data set; the other part is other small crack propagation trace missing region data sets marked by semantic segmentation masks in the total data set, and is called small crack propagation trace missing region data sets;
step 2: carrying out a pre-training process: carrying out outer frame bounding box marking on most crack propagation trace missing area data sets by using a labeling tool LabelImg, taking the gray level images of the missing areas marked by the outer frame bounding box, the marked label images and the original images containing the crack propagation trace missing areas as weak supervision training data sets, carrying out weak supervision training on the auxiliary segmentation model until the model converges, and obtaining mask label images of the most crack propagation trace missing area data sets; wherein the auxiliary segmentation model employs an existing encoder-decoder segmentation model;
step 2.1: marking the outer frame bounding box of the image in the data set of the mostly crack propagation trace missing area by using a labeling tool LabelImg to obtain a texture gray image and a label image of the missing area marked by the outer frame bounding box;
step 2.2, taking an original image in a data set of a large part of crack expansion trace missing area, a texture gray image of the missing area marked by an outer frame bounding box and a label image as a weak supervision data set;
step 2.3: selecting a group of weakly supervised training data set samples from the weakly supervised data set, inputting the samples into an auxiliary segmentation model, and carrying out weakly supervised training on the auxiliary segmentation model;
step 2.4: extracting background features of the bounding box label image and image features of the missing region through a segmentation encoder in the auxiliary segmentation model, and learning a logical relation between the background features of the bounding box and the image features of the missing region to obtain a predicted missing region;
step 2.5: calculating the error between the predicted missing region and the background morphology by using a segmentation decoder, adding a self-correction module into the segmentation decoder, and distinguishing and gradually reducing the error between the predicted missing region and the background morphology by using the cross entropy loss in the self-correction module;
step 2.6: repeating the step 2.1-2.5 until the auxiliary segmentation model converges to obtain a mask label image of the data set of the mostly crack propagation trace missing area;
and step 3: the training process is performed as shown in fig. 3: carrying out pixel-level mask marking on a small part of crack propagation trace missing area data set by using a marking tool Labelme to obtain a marked missing area gray image and a mask label image; taking the marked gray level image of the missing area, the mask label image and the original image of the crack expansion trace missing area as a full-supervision training data set; inputting two types of texture gray images and mask label images of the missing region in a data set of the missing region of most crack extension traces and a data set of the missing region of small crack extension traces into a semantic segmentation model of the missing region for semi-supervised training until the model converges to obtain accurate prediction missing regions of the data set of the missing region of most crack extension traces and the data set of the missing region of small crack extension traces; wherein, a FasterR-CNN model is adopted in the semantic segmentation model of the deletion region;
the steps of training the semantic segmentation model of the missing region through the data set of the missing region of the small crack propagation trace are as follows:
step 3.1: carrying out pixel-level mask marking on the image in the data set of the small crack propagation trace missing area by using a marking tool Labelme to obtain a marked missing area gray image and a mask label image;
step 3.2: taking an original image of the crack propagation trace missing area, a gray image of the crack propagation trace missing area marked by a marking tool Labelme and a mask label image as a fully supervised training dataset;
step 3.3: selecting a group of fully supervised training data set samples from a fully supervised training data set, inputting the samples into a missing region semantic segmentation model, and training the missing region semantic segmentation model;
step 3.4: extracting background morphology and image characteristics of the missing region through a Resnet50 network in the semantic segmentation model of the missing region to obtain a shared characteristic layer; acquiring a suggestion frame by using the shared characteristic layer, decoding the suggestion frame, and intercepting the position with the missing region at the shared characteristic layer to obtain a predicted missing region;
step 3.5: calculating the coincidence degree of all the suggestion frames and the real frames by using an ROI module in the semantic segmentation model of the missing region, screening, finally adjusting and convolving, and adjusting the suggestion frames into the real frames to obtain an accurate prediction missing region;
step 3.6: repeating the steps of 3.1-3.5 until the training of the data set of the small crack propagation trace missing region is finished, and converging the semantic segmentation model of the missing region;
the steps of training the missing region segmentation model through the data set of the missing region of most crack propagation traces are as follows:
step 3.7: inputting the mask label image and the original image of the crack expansion trace missing area, which are obtained by the missing area image marked by the bounding box in the step 2 through the auxiliary segmentation model, into the missing area semantic segmentation model;
step 3.8: the rest steps are the same as the steps 3.3-3.4, until the training of most crack propagation trace missing area data sets is finished, the semantic segmentation model of the missing area is converged;
and 4, step 4: fusing most crack propagation trace missing region data set images and small crack propagation trace missing region data set images which are input and trained by a missing region semantic segmentation model by using a multi-scale feature combination method, and training a fine contour model based on a generated countermeasure network; wherein, the fine contour model based on the generation countermeasure network adopts GAN generation countermeasure network;
step 4.1: taking the original image of the crack expansion trace missing area in the step 2 and the step 3, and the gray level image and the Mask label image which are obtained respectively as training data sets, and training a fine contour model based on the generated countermeasure network;
step 4.2: selecting a group of training data sets in the step 4.1 as samples, and inputting the samples into a fine contour model based on the generated countermeasure network;
step 4.3: extracting the contour features of the background feature and the missing region based on a generator in a fine contour model of a generated countermeasure network, and learning the logical relationship between the background feature and the contour features of the missing region to obtain a predicted missing contour;
step 4.4: calculating the error of the predicted missing contour and the real original contour by using a discriminator in a fine contour model for generating the countermeasure network, distinguishing the predicted missing contour and the real original contour, and optimizing a generator;
step 4.5: repeating the steps 4.3-4.4 until the fine contour model based on the generated countermeasure network converges;
in this embodiment, the GAN generation countermeasure network includes a generator g (generator) and a discriminator d (discriminator). The generator randomly extracts samples from the noise distributed a priori, obtains corresponding generated samples through the generator, and distinguishes the two samples as much as possible between the real sample and the generated sample through the discriminator. After the discriminator is updated for k times in a circulating manner, the parameters of the generator are updated once by using a smaller learning rate, and the generator is trained to reduce the difference between the generated sample and the real sample as much as possible, which is also equivalent to making the discriminator wrong in discrimination as much as possible. After a number of update iterations, the final ideal case is for the discriminator to be unable to discriminate whether the sample is coming from the output of the generator or the true output.
In the embodiment, a texture gray image and a Mask label image of a marked missing region are used as a training data set, and a fine contour model based on a generated countermeasure network is trained; the GAN generation countermeasure network extracts the background morphology and the outline characteristics of the missing area through a generator. Learning a logical relation between the background morphology and the outline characteristics of the missing region to obtain a predicted missing outline; and calculating the error between the predicted missing contour and the real original contour by using a discriminator, learning and distinguishing the predicted missing contour and the real original contour, and optimizing a generator.
The generator integrates knowledge distillation learning mechanisms of model distillation and data distillation to improve the network training environment, and as shown in fig. 4, a deep inference network based on multi-source memory and learning of a representative crack propagation characteristic contour is established to realize high-precision representation of the contour.
The model distillation method adopts a training multisource comprehensive network to fit the output of a plurality of single-source contour generation networks, introduces a loss function of an original single-source network into the multisource comprehensive network, aligns the contour output by the multisource comprehensive network with the single-source network in a mapping way, evaporates a non-associated data structure, and outputs contour textures with abstract associated characteristics.
The data distillation method trains an initial deep inference network through synchronous iteration of multi-source data, optimizes inference capability of a comprehensive network through re-iteration of the multi-source data, obtains pseudo labels of profile data in an integrated mode, and synchronously trains a deep inference network model of the profile by using supervision data and pseudo labels without supervision data.
The discriminator is constructed by the countermeasure loss and the characteristic matching loss together, and is defined as:
Figure BDA0003368970470000081
wherein λ isa1、λfA weight parameter that is a loss function;
the challenge loss is defined as:
Figure BDA0003368970470000082
wherein, PrealIs a real outline picture; grealIs a real gray scale image; ppredA prediction result for the generator; e represents expectation;
the feature matching penalty achieves a stable training process by having the generator produce results with similarities to real images to generate discriminators defined as:
Figure BDA0003368970470000083
wherein L is the number of convolution layers of the discriminator; n is a radical ofiThe activation result of the ith layer of the discriminator;
and 5: and (3) carrying out a segmentation acquisition process: taking the original image with the missing crack propagation trace, the gray image with the missing crack propagation trace and the Mask label image as input images, and performing alternate iteration and optimization on the generator and the discriminator by using the fine contour model trained in the step 4 and based on the generation countermeasure network to finally obtain a fine segmentation image of the missing area;
step 5.1: carrying out bounding box marking on the image containing the crack propagation trace missing area according to the step 2 to obtain a texture gray image and a label image of the missing area;
step 5.2: performing feature extraction on the texture gray level image and the label image of the missing region in the step 5.1 by using the auxiliary segmentation model trained in the step 2 to obtain an accurate mask label image;
step 5.3: inputting an original image containing a crack expansion trace missing area, a texture gray level image of the missing area and a mask label image into a fine contour model based on a generated countermeasure network;
step 5.4: extracting multi-scale background morphology and missing region contour characteristics based on a generator in a fine contour model for generating a countermeasure network, predicting a missing contour, and realizing reconstruction and restoration of the contour in the missing region;
step 5.5: calculating errors of the predicted missing contour and the real original contour by using a discriminator in a fine contour model based on a generated confrontation network, performing confrontation loss training, reducing the errors of the missing contour and the real contour, and optimizing a generator;
step 5.6: and finally obtaining a complete refined crack propagation characteristic segmentation contour by combining the characteristic contour of the original image and the repaired characteristic contour in the missing area.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions and scope of the present invention as defined in the appended claims.

Claims (6)

1. A method for segmenting crack propagation trace missing regions based on deep learning is characterized by comprising the following steps: preprocessing an image containing a crack propagation trace missing region, firstly, utilizing a marking tool to mark a bounding box image of a part of a crack propagation trace missing region data set image, then inputting the image into an auxiliary segmentation model to perform weak supervision training to obtain an accurate mask semantic segmentation image, and inputting the mask semantic segmentation image into a missing region semantic segmentation model; simultaneously, the other part of the crack propagation trace missing area data set image is subjected to mask marking by using a marking tool and then is input into a missing area semantic segmentation model for full supervision training; fusing a part of crack propagation trace missing region data set images input into the missing region semantic segmentation model and trained with another part of crack propagation trace missing region data set images by using a multi-scale feature combination method to obtain missing segmentation regions; and finally, forming a fine contour segmentation model of the crack propagation trace missing region by utilizing a game mechanism for generating the countermeasure network, and accurately segmenting the crack propagation trace missing region of the input picture.
2. The deep learning-based crack propagation trace missing region segmentation method according to claim 1, characterized in that: the method specifically comprises the following steps:
step 1: acquiring an image containing a crack propagation trace missing area to form a data set; the data set is divided into two parts: one part of the crack propagation trace missing region data set accounts for more than seventy percent of the total data set and is used for marking the outer frame bounding box, and the data set is called as a most part of crack propagation trace missing region data set; the other part is other crack propagation trace missing region data sets marked by semantic segmentation masks in the total data set, and is called a small part crack propagation trace missing region data set;
step 2: carrying out a pre-training process: carrying out outer frame bounding box marking on most crack propagation trace missing area data sets by using a marking tool, taking the gray level images of the missing areas marked by the outer frame bounding box, the marked label images and the original images containing the crack propagation trace missing areas as weak supervision training data sets, carrying out weak supervision training on the auxiliary segmentation model until the model converges, and obtaining mask label images of most crack propagation trace missing area data sets; wherein the auxiliary segmentation model employs an existing encoder-decoder segmentation model;
and step 3: carrying out a training process: carrying out pixel-level mask marking on a small part of crack propagation trace missing area data set by using a marking tool to obtain a marked missing area gray image and a mask label image; taking the marked gray level image of the missing area, the mask label image and the original image of the crack expansion trace missing area as a full-supervision training data set; inputting two types of texture gray images and mask label images of the missing region in a data set of the missing region of most crack extension traces and a data set of the missing region of small crack extension traces into a semantic segmentation model of the missing region for semi-supervised training until the model converges to obtain accurate prediction missing regions of the data set of the missing region of most crack extension traces and the data set of the missing region of small crack extension traces; wherein, the missing region semantic segmentation model adopts a Faster R-CNN model;
and 4, step 4: fusing most crack propagation trace missing region data set images and small crack propagation trace missing region data set images which are input and trained by a missing region semantic segmentation model by using a multi-scale feature combination method, and training a fine contour model based on a generated countermeasure network; wherein, the fine contour model based on the generation countermeasure network adopts GAN generation countermeasure network;
and 5: and (3) carrying out a segmentation acquisition process: and (4) taking the original image with the missing crack propagation trace, the gray image with the missing crack propagation trace and the Mask label image as input images, and performing alternate iteration and optimization on the generator and the discriminator by using the fine contour model trained in the step (4) based on the generation countermeasure network to finally obtain a fine segmentation image of the missing region.
3. The deep learning-based crack propagation trace missing region segmentation method according to claim 2, characterized in that: the specific method of the step 2 comprises the following steps:
step 2.1: marking the outer frame bounding box of the image in the data set of the most crack propagation trace missing area by using a marking tool to obtain a texture gray image and a label image of the missing area marked by the outer frame bounding box;
step 2.2, taking an original image in a data set of a large part of crack expansion trace missing area, a texture gray image of the missing area marked by an outer frame bounding box and a label image as a weak supervision data set;
step 2.3: selecting a group of weakly supervised training data set samples from the weakly supervised data set, inputting the samples into an auxiliary segmentation model, and carrying out weakly supervised training on the auxiliary segmentation model;
step 2.4: extracting background features of the bounding box label image and image features of the missing region through a segmentation encoder in the auxiliary segmentation model, and learning a logical relation between the background features of the bounding box and the image features of the missing region to obtain a predicted missing region;
step 2.5: calculating the error between the predicted missing region and the background morphology by using a segmentation decoder, adding a self-correction module into the segmentation decoder, and distinguishing and gradually reducing the error between the predicted missing region and the background morphology by using the cross entropy loss in the self-correction module;
step 2.6: and (5) repeating the steps 2.1-2.5 until the auxiliary segmentation model is converged to obtain a mask label image of the data set of the mostly crack propagation trace missing area.
4. The deep learning-based crack propagation trace missing region segmentation method according to claim 3, characterized in that: step 3, training the semantic segmentation model of the missing region through the data set of the missing region of the small crack propagation trace, and the steps are as follows:
step 3.1: carrying out pixel-level mask marking on the image in the data set of the small crack propagation trace missing area by using a marking tool Labelme to obtain a marked missing area gray image and a mask label image;
step 3.2: taking an original image of the crack propagation trace missing area, a gray image of the crack propagation trace missing area marked by a marking tool Labelme and a mask label image as a fully supervised training dataset;
step 3.3: selecting a group of fully supervised training data set samples from a fully supervised training data set, inputting the samples into a missing region semantic segmentation model, and training the missing region semantic segmentation model;
step 3.4: extracting background morphology and image characteristics of the missing region through a Resnet50 network in the semantic segmentation model of the missing region to obtain a shared characteristic layer; acquiring a suggestion frame by using the shared characteristic layer, decoding the suggestion frame, and intercepting the position with the missing region at the shared characteristic layer to obtain a predicted missing region;
step 3.5: calculating the coincidence degree of all the suggestion frames and the real frames by using an ROI module in the semantic segmentation model of the missing region, screening, finally adjusting and convolving, and adjusting the suggestion frames into the real frames to obtain an accurate prediction missing region;
step 3.6: repeating the steps of 3.1-3.5 until the training of the data set of the small crack propagation trace missing region is finished, and converging the semantic segmentation model of the missing region;
the steps of training the missing region segmentation model through the data set of the missing region of most crack propagation traces are as follows:
step 3.7: inputting the mask label image and the original image of the crack expansion trace missing area, which are obtained by the missing area image marked by the bounding box in the step 2 through the auxiliary segmentation model, into the missing area semantic segmentation model;
step 3.8: the rest steps are the same as the steps 3.3-3.4, until the training of most crack propagation trace missing area data sets is finished, and the semantic segmentation model of the missing area is converged.
5. The deep learning-based crack propagation trace missing region segmentation method according to claim 4, characterized in that: the specific method of the step 4 comprises the following steps:
step 4.1: taking the original image of the crack expansion trace missing area in the step 2 and the step 3, and the gray level image and the Mask label image which are obtained respectively as training data sets, and training a fine contour model based on the generated countermeasure network;
step 4.2: selecting a group of training data sets in the step 4.1 as samples, and inputting the samples into a fine contour model based on the generated countermeasure network;
step 4.3: extracting the contour features of the background feature and the missing region based on a generator in a fine contour model of a generated countermeasure network, and learning the logical relationship between the background feature and the contour features of the missing region to obtain a predicted missing contour;
step 4.4: calculating the error of the predicted missing contour and the real original contour by using a discriminator in a fine contour model for generating the countermeasure network, distinguishing the predicted missing contour and the real original contour, and optimizing a generator;
step 4.5: and repeating the steps 4.3-4.4 until the fine contour model based on the generated countermeasure network converges.
6. The deep learning-based crack propagation trace missing region segmentation method according to claim 5, characterized in that: the specific method of the step 5 comprises the following steps:
step 5.1: carrying out bounding box marking on the image containing the crack propagation trace missing area according to the step 2 to obtain a texture gray image and a label image of the missing area;
step 5.2: performing feature extraction on the texture gray level image and the label image of the missing region in the step 5.1 by using the auxiliary segmentation model trained in the step 2 to obtain an accurate mask label image;
step 5.3: inputting an original image containing a crack expansion trace missing area, a texture gray level image of the missing area and a mask label image into a fine contour model based on a generated countermeasure network;
step 5.4: extracting multi-scale background morphology and missing region contour characteristics based on a generator in a fine contour model for generating a countermeasure network, predicting a missing contour, and realizing reconstruction and restoration of the contour in the missing region;
step 5.5: calculating errors of the predicted missing contour and the real original contour by using a discriminator in a fine contour model based on a generated confrontation network, performing confrontation loss training, reducing the errors of the missing contour and the real contour, and optimizing a generator;
step 5.6: and finally obtaining a complete refined crack propagation characteristic segmentation contour by combining the characteristic contour of the original image and the repaired characteristic contour in the missing area.
CN202111392850.7A 2021-11-23 2021-11-23 Deep learning-based crack propagation trace missing region segmentation method Pending CN114066867A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111392850.7A CN114066867A (en) 2021-11-23 2021-11-23 Deep learning-based crack propagation trace missing region segmentation method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111392850.7A CN114066867A (en) 2021-11-23 2021-11-23 Deep learning-based crack propagation trace missing region segmentation method

Publications (1)

Publication Number Publication Date
CN114066867A true CN114066867A (en) 2022-02-18

Family

ID=80279248

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111392850.7A Pending CN114066867A (en) 2021-11-23 2021-11-23 Deep learning-based crack propagation trace missing region segmentation method

Country Status (1)

Country Link
CN (1) CN114066867A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114758125A (en) * 2022-03-31 2022-07-15 江苏庆慈机械制造有限公司 Gear surface defect detection method and system based on deep learning
CN114894642A (en) * 2022-07-01 2022-08-12 湖南大学 Fatigue crack propagation rate testing method and device based on deep learning

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114758125A (en) * 2022-03-31 2022-07-15 江苏庆慈机械制造有限公司 Gear surface defect detection method and system based on deep learning
CN114894642A (en) * 2022-07-01 2022-08-12 湖南大学 Fatigue crack propagation rate testing method and device based on deep learning
CN114894642B (en) * 2022-07-01 2023-03-14 湖南大学 Fatigue crack propagation rate testing method and device based on deep learning

Similar Documents

Publication Publication Date Title
CN111582294B (en) Method for constructing convolutional neural network model for surface defect detection and application thereof
CN114066867A (en) Deep learning-based crack propagation trace missing region segmentation method
WO2023207104A1 (en) Ground penetrating radar tunnel lining quality inspection method based on self-supervised learning
CN111178392B (en) Aero-engine hole detection image damage segmentation method based on deep neural network
CN112884758B (en) Defect insulator sample generation method and system based on style migration method
CN114049343A (en) Deep learning-based tracing method for complex missing texture of crack propagation process
Tian et al. CASDD: Automatic surface defect detection using a complementary adversarial network
CN114898472B (en) Signature identification method and system based on twin vision transducer network
CN114758329A (en) System and method for predicting temperature of target area in thermal imaging graph based on deep learning
CN115564031A (en) Detection network for glass defect detection
Fan et al. Application of YOLOv5 neural network based on improved attention mechanism in recognition of Thangka image defects
CN117437647B (en) Oracle character detection method based on deep learning and computer vision
CN113077438B (en) Cell nucleus region extraction method and imaging method for multi-cell nucleus color image
CN117058266B (en) Handwriting word generation method based on skeleton and outline
CN114581932A (en) Picture table line extraction model construction method and picture table extraction method
CN117540779A (en) Lightweight metal surface defect detection method based on double-source knowledge distillation
CN116110047A (en) Method and system for constructing structured electronic medical record based on OCR-NER
CN113192018B (en) Water-cooled wall surface defect video identification method based on fast segmentation convolutional neural network
CN115147303A (en) Two-dimensional ultrasonic medical image restoration method based on mask guidance
CN114694019A (en) Remote sensing image building migration extraction method based on anomaly detection
Balzategui et al. Few-shot incremental learning in the context of solar cell quality inspection
CN113191967A (en) Thangka image comprehensive restoration method
CN116993721B (en) Steel plate surface defect detection method based on weak supervision neural network
CN112329603B (en) Dam face crack defect positioning method based on image cascade
CN112329863B (en) Method for identifying state of isolation switch in traction substation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination