CN117496003B - Defect image generation method of industrial element - Google Patents

Abstract

The invention relates to the technical field of image processing, solves the technical problem that model training requirements cannot be met due to scarcity of defective picture sample data, in particular to a defective image generation method of an industrial element, which comprises the following steps: s1, pre-training based on INPAINTING METHOD models to obtain pre-training weights of an image processing model Unet; s2, training the image processing model Unet by adopting pre-training weights and images under different scenes to generate a defect image generation model; s3, importing the pure background image of the defect to be generated into a defect image generation model to finish image generation of specified defect details. The invention is motivated by the open source model INPAINTING METHOD and employs a number of creative designs so that the type, location, and shape of the generated defects can be specified from the front end or extracted from an existing database.

Description

Defect image generation method of industrial element

Technical Field

The invention relates to the technical field of image processing, in particular to a defect image generation method of an industrial element.

Background

With the continuous development of intelligent technology, the combination of industrial production and digitization is gradually deepened, and more industries begin to utilize the emerging intelligent detection means based on machine vision at the upstream and downstream of the manufacturing process, gradually replace the traditional but inefficient manual detection, and exercise the responsibility of detecting defective elements. The intelligent detection method rely on of the machine vision is a well-trained model, so that the intelligent detection method cannot meet expected requirements due to under fitting and over fitting in practical application. To do this, a large number of defect samples, balanced against the non-defective samples, are required for training to reflect the general characteristics of the defects.

However, in industrial practice, as the manufacturing level is continuously improved, the production technology is becoming more and more exquisite, defect data is becoming more and more scarce, and the problem of unbalanced distribution of defect types in a defect data set is represented by long tail distribution problem, so that model training is more difficult, and in order to cope with the dilemma, how to obtain a large amount of data reflecting corresponding details of different defect types becomes a challenge.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a defect image generation method of an industrial element, which solves the technical problem that the model training requirement cannot be met due to the scarcity of defect picture sample data.

In order to solve the technical problems, the invention provides the following technical scheme: a method of generating a defective image of an industrial component, the method comprising the steps of:

S1, pre-training based on INPAINTING METHOD models to obtain pre-training weights of an image processing model Unet;

s2, training the image processing model Unet by adopting pre-training weights and images under different scenes to generate a defect image generation model;

S3, importing the pure background image of the defect to be generated into a defect image generation model to finish image generation of specified defect details.

Further, in step S1, the specific process includes the following steps:

S11, manufacturing training images: any picture with industrial defects and the corresponding label of the picture are read in and converted into a Tensor format, and the picture is stored as a group_ truth picture;

s12, defining a training area: according to segmentation representing defect segmentation and category representing defect types in the section corresponding to each group_ truth picture, making a condition guidance picture condition_map with the same size as the group_ truth picture, superposing the group_ truth picture and the condition guidance picture condition_map to generate a six-dimensional input picture, and then according to the section corresponding to the group_ truth picture, representing a boundary_box with unknown defects as a defined training area;

S13, training: and manufacturing a mask representing the defective area according to the marking_box, and training the area representing the defect in windowed _input to obtain a pre-training weight.

Further, in step S2, the images under different scenes include images with defect labels and pure background images without defect labels.

Further, in step S3, specifying defect details includes specifying the position, type, and shape of the defect in the pure background image.

Further, in step S2, the image processing model is trained by using the image with the defect label, and the specific process includes the following steps:

s211, manufacturing training images: reading in a picture, performing data enhancement on the picture excluding 'restore' and 'random_crop', converting the picture into a Tensor format, and storing the picture as a group_ truth picture;

S212, demarcating a training area: reading a detection frame in the picture mark, and giving a mask representing a training area through the detection frame;

S213, creating a conditional_map: reading in a digital index representing the type of defect contained in the picture, and then converting the index into a color representing the type of defect through a pre-defined Python dictionary;

Reading in semantic segmentation points boundary_points representing the edges of the defects contained in the picture, filling colors on the inner walls defined by the edges and the edges by utilizing colors representing the types of the defects, attenuating the filling and the edges according to a rejection rate drop_rate of 70%, covering the picture segmentation representing the points of the defects after the attenuation, and adding a conditional_map as a conditional_map;

S214, determining training window input: determining the part of the original image which is input into a network for extracting defect information according to a detection box of the defect contained in the image read in previously, performing slicing operation on the original image and a conditional_map of the conditional guide image according to the window, obtaining an image which is not scaled, overlapping the image into a six-dimensional initial feature image, and then sending the six-dimensional initial feature image into an image processing model for training;

S215, manufacturing an expansion_mask and a shrnk_mask by adopting a MASK IN MASK method: according to the track_mask representing the training area received by the image processing model, scaling the length and the width of the track_mask to manufacture an expansion_mask and a shrank_mask according to a preset ratio, wherein the expansion_mask is slightly larger than the track_mask, and the shrank_mask is slightly smaller than the track_mask;

randomly taking one of expansion_mask and spring_mask as a training area, and taking the rest content including defect foreground information as generated prompt characteristic information;

s216, selecting nonlinear attenuation during training: when the image processing model is trained, a large attenuation value is used in the small step number, and a small attenuation value is used in the opposite way;

S217, training the image processing model is finished to generate a defect image generation model, and a training result is evaluated.

Further, in step S2, the image processing model is trained by using a pure background image without defect labeling, and the specific process includes the following steps:

S221, manufacturing training images: reading in a picture, performing data enhancement on the picture excluding 'restore' and 'random_crop', converting the picture into a Tensor format, and storing the picture as a group_ truth picture;

S222, determining a generation area: randomly generating a detection box marking-box representing a generation area, and obtaining a mask representing a background area to be learned in a range defined by the detection box marking-box;

S223, preparing a condition guide chart: the background area to be learned represented by the mask is blacked on the conditional_mask of the conditional guidance graph;

S224, determining training window input: determining the part of the original image which is input into the network for extracting defect information according to a detection box of the defect contained in the image read previously, performing slicing operation on the original image and the conditional_map of the conditional guide image according to the window, obtaining the image which is not scaled, overlapping the image into a six-dimensional initial characteristic image, and then sending the six-dimensional initial characteristic image into an image processing model for training, namely overlapping the original image of the window and the conditional_map of the conditional guide image as input;

S225, iterative training: and (5) evaluating the result until the specified steps are completed.

Further, in step S3, the specific process includes the following steps:

S31, obtaining a background target picture: reading a pure background picture which needs to generate defects, converting the pure background picture into a Tensor format, and storing the pure background picture as a background picture;

S32, obtaining a target guide diagram: taking out from the existing database, or manually designating a condition guide map, which can represent the type, position and shape detail information of the defect to be generated, and superposing the detail information with the background picture to generate an original feature map, and determining a generation_mask representing a designated generation area according to the condition guide map;

S33, obtaining window input: determining a part of the original feature map, which is input into a network for generating defect information, according to the generated mask of the designated generation area, performing slicing operation on the original feature map according to the window to obtain a feature map which is not scaled, and determining a mask which needs to be generated in the window;

s34, manufacturing an expansion_mask and a shrnk_mask by adopting a MASK IN MASK method: manufacturing an expansion_mask and a krink_mask according to windowed _generate_mask representing a training area received by a defect image generation model through designating a ratio;

S35, generating defect contents according to the steps: firstly reconstructing defect content in a spring_mask to obtain feature information of defect prompt in an expansion_mask range;

reconstructing defect content in the expansion_mask to obtain feature information of defect prompt in the shrnk_mask;

S36, finishing defect generation and pasting defects: after the defect feature generation is completed, the window for generating is pasted back into the original image, and the defect generation flow is completed.

By means of the technical scheme, the invention provides a defect image generation method of an industrial element, which has at least the following beneficial effects:

1. When the defect image generation model provided by the invention reads the image containing the defect foreground, the conditional guidance image of the defect foreground attached to the image is stored in the database, so that the defect image generation model can be used as a guide when the defect image generation model generates the image containing the defect foreground, the practicability of the defect image generation model is improved, the application scene can be widened, and the defect foreground characteristic which is closer to the actual logic defect shape is generated.

2. The generating method provided by the invention focuses on the foreground characteristics of the defects, and simultaneously focuses on the background information equally by fine adjustment on the conditional guide graph, so that the background information of the generated region and the background information of the non-generated region do not have obvious boundaries, the tone is more uniform, the reality logic is more met, and meanwhile, new information is introduced with probability, so that the background richness of the generated pictures is increased.

3. The invention provides a method for specifying specific details of defects through a conditional guidance graph, so that the details such as the position, the type, the shape and the like of the defects can be manually determined from the front end, a user is given great autonomy, more pictures containing different details, different positions and different types of defects can be generated, the application scene is wider, and the richness of a data set is greatly increased.

4. The invention adopts a characteristic input mode based on a sliding window, so that the characteristic image obtained by the model is an original image which is not subjected to large-scale, thus the serious information loss problem caused by large-scale scaling is avoided, the defect foreground and background information with more details can be obtained during training, and the generated image is more in accordance with the reality logic.

5. According to the invention, a MASK IN MASK method (MIM) is adopted to divide a generation area into an expansion_mask and a kringle_mask, and training is carried out respectively in a training time zone, when defects are generated, the characteristics of defect information prompt in the expansion_mask are used for generating the defect information in the kringle_mask, and then the characteristic information of defect prompt generated in the kringle_mask is used for generating the defect information and a part of background information in the expansion_mask.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

FIG. 1 is a flow chart of a defect image generating method of the present invention;

FIG. 2 is a diagram showing an example of a conditional_map representing a training area according to the present invention;

FIG. 3 is an exemplary diagram of generating a complete conditional_map in accordance with the present invention;

FIG. 4 is an exemplary diagram of determining training window inputs in accordance with the present invention;

FIG. 5 is an exemplary diagram of the present invention for creating an expand_mask and a shrnk_mask;

FIG. 6 is an exemplary plot of nonlinear attenuation in the selection training of the present invention;

fig. 7 is a comparison of a defective picture with an original picture according to the present invention.

Detailed Description

In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description. Therefore, the realization process of how to apply the technical means to solve the technical problems and achieve the technical effects can be fully understood and implemented.

Those of ordinary skill in the art will appreciate that all or a portion of the steps of a method in an implementation may be accomplished by programming instructions to the associated hardware, and thus the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

With the continuous development of intelligent technology, the combination of industrial production and digitization is gradually deepened, and more industries begin to utilize the emerging intelligent detection means based on machine vision at the upstream and downstream of the manufacturing process, gradually replace the traditional but inefficient manual detection, and exercise the responsibility of detecting defective elements. The intelligent detection method rely on of the machine vision is a well-trained model, so that the intelligent detection method cannot meet expected requirements due to under fitting and over fitting in practical application. To do this, a large number of defect samples, balanced against the non-defective samples, are required for training to reflect the general characteristics of the defects. However, in industrial practice, as the manufacturing level is continuously improved, the production technology is becoming more and more exquisite, the defect data is becoming more and more scarce, and the problem of unbalanced distribution of each defect type in the defect data set is represented by long tail distribution problem, so that model training is more difficult. To cope with this dilemma, it is a challenge how to obtain a large amount of data reflecting the corresponding details of the different defect categories. As the industry continues to dig into this problem, a large array of solutions have emerged that are of practical interest, including manual solutions as well as generative solutions.

Traditional manual schemes, i.e. creating a defect area database on top of a large number of defect pictures collected. The defect areas selected according to the requirements are pasted on normal samples without defects, and background fusion including poisson fusion is carried out if necessary, so that the pasting areas are unified with the original background, a large number of defect samples are manufactured, and the problem of scarcity of the number of defect pictures is solved.

The generation scheme is that an image generation network is used, after the defect characteristics are fully learned, a defect area is generated on a normal sample without defects, so that a large number of defect samples are generated, and the problem of scarcity of the number of defect images is solved. In the field of image generation, a diffusion model using DDPM as a basic module, a generation model using GAN as a basic module and a Encoder-Decoder generation model both have good image generation capability, and are common choices for solving the problem of scarcity of a defect sample.

Therefore, the generation of the picture for the defect is mainly disadvantageous in that:

The conventional general generation scheme cannot specify the type of the generated defect, or needs to input a picture with the same defect type, so as to reconstruct a picture containing the defect type. Also in the picture, the position of the generated defect is difficult to be precisely specified. In the same defect type, the shapes of defects are also obviously different, for example, the trend of defects in different pictures is obviously different, and a general generation method cannot accurately control the generation trend, so that the generated picture set is poor in richness. Similarly, the defect type 'stain' is the same, in different defect pictures, the size of the space occupied by the 'stain' is not unified standard in different pictures, the size of the defect of the generated stain cannot be accurately specified by a general generation model, and the diversity of the generated defect data set pictures is limited.

The existing method for artificially manufacturing the defect image generally faces the dilemma that the boundary between the generated area and the non-generated area is too obvious, and the tone and detail of the background in the generated area and the lack of consistency of the rest area in the picture lead to the defect picture which is artificially generated not to conform to the actual logic, has a gap with the actual defect picture and lacks training value.

The input port of the deep learning network and the size of the generated image which are relied on by the conventional general image generation scheme are often smaller, so that the image must undergo large-scale scaling before being input into a model, so that a great amount of semantic information of background and foreground defects is lost, training cannot be well assisted, and the real characteristic condition is reflected. Meanwhile, the generated image also faces the problems that the size is too small, the color tone is too small, the feature contained in the background and the defect foreground is too little, and the feature does not accord with the reality logic, so that the image does not have the trained significance, has a gap with the real defect picture, and lacks training value.

The existing method for manually manufacturing the defect image is focused on extracting the characteristic information of the defect, but cannot restore background information of non-foreground in a designated area, such as undamaged parts in the designated area. The method ensures that the logic of the manufactured image has no good logic continuity between the non-designated area and the designated area, and also ensures that the background information of the non-foreground generated by the designated area is different from the real image.

Based on the technical problems of the prior art, please refer to fig. 1-7, which illustrate a specific implementation of the present embodiment, the present embodiment is inspired by the open source model INPAINTING METHOD, and a large number of creative designs are adopted, so that the types, positions and shapes of the generated defects can be specified from the front end or extracted from the existing database. The model reads in the pictures with defects in three different execution scenes, namely, reads in the pure background pictures without defects in training and generates the defect pictures in training, and the specific steps adopted in the three scenes are different, but the user can specify the details of generating the defects, and the model has better background characteristic information restoration capability and less characteristic information loss, so that the model learns the background characteristic information while focusing on the defect foreground information, a part of background information can be introduced while the model generates the foreground information, and the network input port is aligned to the defect area in training, so that a large amount of information is not required to be lost by zooming the pictures.

In order to achieve the above objective, referring to fig. 1, the present embodiment provides a method for generating a defect image of an industrial device, which includes the following steps:

S1, pre-training based on INPAINTING METHOD models to obtain pre-training weights of an image processing model Unet; in step S1, the specific process includes the following steps:

S11, manufacturing training images: any picture with industrial defects and the corresponding label of the picture are read in and converted into a Tensor format and stored as a group_ truth picture, and in the step, the picture which does not accord with the input size of a network window does not need to be scaled so as to avoid information loss.

S12, defining a training area: according to segmentation representing defect segmentation and category representing defect type in the section corresponding to each group_ truth picture, a conditional_map of the same size as the group_ truth picture is produced, the group_ truth picture and the conditional_map are overlapped to generate a six-dimensional input picture, and then according to the section corresponding to the group_ truth picture, a training area is defined, namely, according to the center coordinates of the group_box, an input area which takes the coordinates as the center is obtained, and only a part of the input area is intercepted and is called windowed _input. Windowed _input is then fed into the image processing model Unet (optionally: pre-training weights using INPAINTING METHOD open source model) for training; an example diagram is shown in fig. 2, with specific pseudocode as follows:

s13, training: according to the mask of the marking box, training the area representing the defect in windowed _input to obtain the pre-training weight, calculating the loss value is only performed in the area represented by the mask, in this step, windowed _input with a fixed length and width of 256x and larger than the area represented by the marking box is usually obtained, as an illustration, for the training process, the loss value of each step is gradually added in the mask area until 1000 steps are finished, the calculation formulas of gradually added noise and calculating the loss value of each step are redundant and complex, but in the relevant literature information of the diffusion model, such as at sections 3.1 and 3.2 of the published literature, the address of the published literature is: https:// arxiv.

In this step, the pre-training is performed on the INPAINTING METHOD model of the open source mainly by using the existing conventional training method, so in the above method step, the pre-training of the INPAINTING METHOD model to obtain the image processing model is a conventional technical means, and also can be understood as directly using the INPAINTING METHOD model of the existing open source, and performing the preliminary training on the model by using the self weight to achieve the purpose of being able to be directly applied, as a disclosure, the address of the INPAINTING METHOD model of the open source adopted in this embodiment is:

GitHub-Janspiry/Palette-Image-to-Image-Diffusion-Models:Unoffi cialimplementation of Palette:Image-to-Image Diffusion Models by Pytorch.

In this embodiment, the disclosure of step S1 can be understood that the pre-training weight of INPAINTING METHOD models is actually downloaded from the open source website, and because the main body of the INPAINTING METHOD model and the defect image generating model proposed in this embodiment is unet of one input six-dimensional picture, the defect image generating model proposed in this embodiment can be trained by using the pre-training weight, which does not represent that a INPAINTING METHOD model must be trained before the defect image generating model is trained. Theoretically, given enough training samples, the defect image generation model can start from zero, and the ideal effect can be achieved after continuous iterative training.

In this embodiment, the baseline model is Unet and the baseline model with the structure of INPAINTING METHOD model is Unet, so that pre-training weights can be used. Theoretically, however, the pre-training weights of Unet may be used without the need for a transfer learning prior to training the defect image generation model, provided that enough training samples are given.

Therefore, only the pre-training weights of INPAINTING METHOD model are adopted in the present embodiment, inspired by INPAINTING METHOD model. Theoretically, given enough training data, an ideal effect can be achieved even without the pre-training weights of INPAINTING METHOD models.

S2, training the image processing model Unet by adopting pre-training weights and images under different scenes to generate a defect image generation model, wherein the images under different scenes comprise images with defect labels and pure background images without the defect labels, and the pre-training weights are obtained in the step S1; in step S2, training the image processing model by using the image with the defect label, and the specific process includes the following steps:

s211, manufacturing training images: reading in a picture, performing data enhancement on the picture excluding 'restore' and 'random_crop', converting the picture into a Tensor format, and storing the picture as a group_ truth picture;

S212, demarcating a training area: reading in a detection frame in the picture annotation, and giving a mask representing a training area through the detection frame, so that the image processing model can be noisy only in the mask during training, and meanwhile, loss is only calculated in the mask, and a pseudo code for generating the mask is specifically as follows:

xmin,ymin,w,h＝bounding_box

mask＝torch.zeros_like(ground_truth)

mask[:,ymin:ymin+h,xmin:xmin+w]＝1

In this embodiment, when focusing on the foreground feature of the defect, the fine adjustment on the conditional guide map also enables the defect image generating model to focus on the background information equally, so that the background information of the generated region and the background information of the non-generated region have no obvious boundary, therefore, the tone is more uniform, the reality logic is more met, and meanwhile, new information is introduced with probability, so that the richness of the background of the generated image is increased.

S213, creating a conditional_map: a digital index representing the type of defect contained in the picture is read in, for example: representing the defect type color by 0, and then converting the index into a color representing the defect type through a pre-defined Python dictionary;

The semantic division point location of the edge representing the defect included in the picture is read in, the color of the edge and the inner wall defined by the edge are filled by the color representing the defect type, the filling and the edge are attenuated according to the reject rate drop rate of 70%, the obtained picture segmentation of the point location representing the defect is covered after attenuation, and the conditional map is added as the conditional map.

The addition of picture segmentation to the conditional_map is herein for convenience of presentation to facilitate reading and understanding of the principle, in actual model estimation, segmentation first experiences nonlinear decay according to the number of steps at each specific step of the model, and the graph conditional_map is guided as a complete condition on addition to the conditional_map. Example figure as shown in figure 3, specific pseudocode details of the operation are as follows:

conditional_map＝deepcoopy(ground_truth)

the area defined by the mask is darkened on the conditional guidance map (mask, 0)

segmentation＝torch.zeros_like(ground_truth)

segmentation＝fill_interior(segmentation,boundary_points)

segmentation＝drop_layer(segmentation,drop_rate＝70％)

conditional_map＝conditional_map+segmentation

In this embodiment, a method for specifying specific details of defects generated by using a conditional guidance graph is provided, so that details such as positions, types, shapes and the like of the generated defects can be manually determined from the front end, great autonomy is given to a user, more pictures containing defects of different details, different positions and different types can be generated, the application scene is wider, and the richness of a data set is greatly increased.

S214, determining training window input: according to the detection box of the defect contained in the picture read in previously, determining the part of the original picture which is input into the network for extracting defect information, carrying out slicing operation on the original picture and the conditional_map of the conditional guide picture according to the window, obtaining the picture which is not scaled, overlapping the picture into a six-dimensional initial characteristic picture, and then sending the six-dimensional initial characteristic picture into an image processing model for training, namely overlapping the original picture of the window and the conditional_map of the window as input. An example diagram is shown in fig. 4, the specific coordinate calculation formula and implementation pseudocode are as follows:

Size of the learning window: window_w, window_h

xmin,ymin,w,h＝bounding_box

Giving the center position of the defect: center_x, center_y=xmin+int (w/2), ymin

+int(h/2)

Learning the upper left corner x coordinate of the window: window_x=int (max (0, center_x-window_w)

/2-1))

Learning the upper left corner y coordinate of the window: window_y=int (max (0, center_y-window_h)

/2-1))

Extracting original pictures through a window: window _ group _ truth = group _ truth [:,

window_y:window_y+window_h,

window_x:window_x+window_w]

Extracting a conditional boot graph through a window: window_conditional_map=

conditional_map[:,

window_y:window_y+window_h,

window_x:window_x+window_w]

The training area window _ mask=window _ mask [: is extracted through the window,

window_y:window_y+window_h,

window_x:window_x+window_w]

Superposing the original window drawing and the window condition guide drawing as input:

input＝concatenate([window_ground_truth,

window_conditional_map],dim＝0)

model.receive(input＝input,train_mask＝window_mask)

in this embodiment, a feature input mode based on a sliding window is adopted, so that a feature map obtained by a model is an original image which is not subjected to large-scale, so that serious information loss problem caused by large-scale scaling is avoided, more detailed defect foreground and background information can be obtained during training, and the generated picture is more in accordance with reality logic.

S215, manufacturing an expansion_mask and a shrnk_mask by adopting a MASK IN MASK method: and (3) manufacturing an expansion_mask and a kringle_mask according to the track_mask representing the training area received by the image processing model and the length and width of the track_mask, wherein the expansion_mask is slightly larger than the track_mask, the kringle_mask is slightly smaller than the track_mask, and the preset ratio is scaled to 70% in the embodiment.

And randomly taking one of the expansion_mask and the spring_mask as a training area, and taking the rest content including defect foreground information as generated prompt characteristic information, so that whether the generated foreground defect or background accords with the real logic. An example diagram is shown in fig. 5, with specific pseudocode as follows:

Mask representing training area: train_mask=model

expand_mask＝get_expand_mask(train_mask,expand_ratio＝ratio)

shrink_mask＝get_shrink_mask(train_mask,shrink_ratio＝ratio)

model.train(input,train_region＝random_choose([expand_mask,

shrink_mask]))

In this embodiment, when a general generation model divides a picture area into a generation area and a non-generation area, the present embodiment originally uses the MASK IN MASK method (MIM) shown above, divides the generation area into an expansion_mask and a kringle_mask, and respectively trains in a training time zone, when generating a defect, uses the feature of defect information prompt in the expansion_mask to generate defect information in the kringle_mask first, and then uses the feature information of defect prompt generated in the kringle_mask to generate defect information in the expansion_mask and a part of background information, and the method for respectively generating and mutually prompting can guide the generated image by taking the existing defect foreground as a part of background, so that the generated defect accords with reality logic better.

S216, selecting nonlinear attenuation during training: when the image processing model is trained, a large attenuation value is used in the small step number, and a small attenuation value is used in the opposite way; in the step, when training, a larger attenuation value is used in a smaller step number, and when the step number is larger, a smaller attenuation value is used to attenuate the conditional guidance graph, so that a certain creativity is exerted when the model can anchor the defect characteristic rapidly. An example is shown in fig. 6, with specific pseudocode as follows:

current_step＝model.get_current_step()

drop_rate＝get_drop_rate(current_step)

conditional_map＝drop_layer(conditional_map,drop_rate)

model.step(conditional_map＝conditional_map)

In this embodiment, a nonlinear attenuation method for feature filling on a conditional guidance graph is innovatively introduced, and according to the method, a larger attenuation coefficient is introduced when the number of steps is smaller, and conversely, a smaller attenuation coefficient is introduced when the number of steps is larger. The defect image generation model can quickly anchor the basic elements of the position, the shape and the category of the generated defect prospect when denoising the Gaussian noise image reversely to generate the defect, unbind the dependence of the conditional guidance image when the generation is gradually completed, and encourage the model to exert creativity.

S217, finishing training the image processing model to generate a defect image generation model, evaluating a training result, evaluating the trained model by using the existing defect picture data set and corresponding labels, reconstructing a defect area at an original defect picture by using the trained model, observing whether the generated defect accords with a reality logic or not by naked eyes, and calculating the FID values of the generated picture and the original picture.

In step S2, training the image processing model by using a pure background image without defect labeling, and the specific process includes the following steps:

S221, manufacturing training images: reading in a picture, performing data enhancement on the picture excluding 'restore' and 'random_crop', converting the picture into a Tensor format, and storing the picture as a group_ truth picture;

S222, determining a generation area: randomly generating a detection box marking_box representing a generation area, and obtaining a mask representing a background area to be learned within the range defined by the detection box marking_box, wherein the specific pseudo code is as follows:

bounding_box＝random_bounding_box()

mask＝random_irregular_mask()

It should be noted that, the mask of the background area to be learned must be within the range framed by the detection box binding_box, that is, ASSERT MASK IN binding_box, "GENERATED MASK out of bound";

s223, preparing a condition guide chart: the background area to be learned represented by the mask is blackened on the conditional_mask of the conditional guidance map, and the pseudo code is as follows:

conditional_map＝deepcopy(ground_truth)

conditional_map.masked_fill_(mask,0)

S224, determining training window input: determining the part of the original image which is input into the network for extracting defect information according to a detection box of the defect contained in the image read previously, performing slicing operation on the original image and the conditional_map of the conditional guide image according to the window, obtaining the image which is not scaled, overlapping the image into a six-dimensional initial characteristic image, and then sending the six-dimensional initial characteristic image into an image processing model for training, namely overlapping the original image of the window and the conditional_map of the conditional guide image as input;

The implementation method of this step is the same as that of step S214, and the specific coordinate calculation formula and implementation pseudo code are the same, and specific contents are referred to each other, and are not described in detail herein.

In this embodiment, a feature input mode based on a sliding window is adopted, so that a feature map obtained by a model is an original image which is not subjected to large-scale, so that serious information loss problem caused by large-scale scaling is avoided, more detailed defect foreground and background information can be obtained during training, and the generated picture is more in accordance with reality logic.

S225, iterative training: until the prescribed number of steps is completed, the evaluation result, in this embodiment, 1000 steps, is obtained, and the evaluation result evaluates whether the generated image conforms to the reality logic by naked eyes, and calculates the FID value between the generated image set and the original image set.

S3, importing the pure background image to be subjected to defect generation into a defect image generation model to finish image generation of specified defect details, wherein the specified defect details comprise positions, types and shapes of specified defects in the pure background image. In step S3, the specific process includes the following steps:

S31, obtaining a background target picture: reading a pure background picture which needs to generate defects, converting the pure background picture into a Tensor format, and storing the pure background picture as a background picture;

S32, obtaining a target guide diagram: and taking out from the existing database or manually designating a condition guide map, which can represent the detail information of the type, position and shape of the defect to be generated, and superposing the detail information with the background picture to generate an original feature map, and determining a generation_mask representing a designated generation area according to the condition guide map. Specifically, the pseudo code is implemented as follows:

conditional_map＝get_conditional_map()

generate_mask＝get_generate_mask(conditional_map)

input＝concatenate([background,conditional_map],dim＝0)

S33, obtaining window input: and determining the part of the original feature map, which is input into the network for generating defect information, according to the generated mask of the designated generation area, performing slicing operation on the original feature map according to the window to obtain a feature map which is not scaled, and determining the mask which needs to be generated in the window. The pseudocode is implemented as follows:

windowed_input＝get_windowed(input,region＝generate_mask)

windowed_generate_mask＝get_windowed(generate_mask,region＝

generate_mask)

model.receive(windowed_input,windowed_generate_mask)

S34, manufacturing an expansion_mask and a shrnk_mask by adopting a MASK IN MASK method: and (3) by designating a ratio of 70%, and manufacturing an expansion_mask and a krink_mask according to windowed _generate_mask representing the training area received by the defect image generation model. The pseudocode is implemented as follows:

expand_mask＝get_expand_mask(windowed_generate_mask,

expand_ratio＝ratio)

shrink_mask＝get_shrink_mask(windowed_generate_mask,

shrink_ratio＝ratio)

S35, generating defect contents according to the steps: firstly reconstructing defect content in a spring_mask to obtain feature information of defect prompt in an expansion_mask range;

reconstructing defect content in the expansion_mask to obtain feature information of defect prompt in the shrnk_mask;

Firstly reconstructing defect contents in the spring_mask, and then reconstructing defect contents in the expansion_mask. When the internal defect of the shrnk_mask is rebuilt, the feature information of the defect prompt in the expansion_mask range can be obtained. And when reconstructing the defect foreground and background in the expansion_mask, obtaining the feature information of the defect prompt in the shrnk_mask. In the number of iterative steps of reconstruction, the nonlinear attenuation in the step S216 attenuates the conditional guidance graph, so that the unbinding is excessively dependent, and the model is encouraged to exert creativity;

S36, finishing defect generation and pasting defects: and respectively walking the predetermined steps on the spring_mask and the expansion_mask, namely respectively walking 1000 steps on the spring_mask and the expansion_mask, and pasting the window for generating back into the original image after the defect feature generation is completed, so as to complete the defect generation flow.

When the defect image generation model provided by the embodiment reads the image containing the defect foreground, the conditional guidance image of the defect foreground attached to the image is stored in the database, so that the defect image generation model can be used as a guide when the defect image generation model generates the picture containing the defect foreground, the practicability of the defect image generation model is improved, the application scene can be widened, and the defect foreground characteristic which is closer to the actual logic defect shape is generated.

Comparison of high quality defective pictures and artwork generated by the defect image generation model as shown in fig. 7, to facilitate comparison of defect generation effects, this example reconstructs defect features on artwork defects, and the conditional guide map uses artwork conditional guide maps.

The foregoing embodiments have been presented in a detail description of the invention, and are presented herein with a particular application to the understanding of the principles and embodiments of the invention, the foregoing embodiments being merely intended to facilitate an understanding of the method of the invention and its core concepts; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims (5)

1. A method for generating a defective image of an industrial component, the method comprising the steps of:

S1, pre-training based on INPAINTING METHOD models to obtain pre-training weights of an image processing model Unet;

s2, training the image processing model Unet by adopting pre-training weights and images under different scenes to generate a defect image generation model;

training an image processing model by adopting an image with a defect label, wherein the specific process comprises the following steps of:

s211, manufacturing training images: reading in a picture, performing data enhancement on the picture excluding 'restore' and 'random_crop', converting the picture into a Tensor format, and storing the picture as a group_ truth picture;

S212, demarcating a training area: reading a detection frame in the picture mark, and giving a mask representing a training area through the detection frame;

S213, creating a conditional_map: reading in a digital index representing the type of defect contained in the picture, and then converting the index into a color representing the type of defect through a pre-defined Python dictionary;

Reading in semantic segmentation points boundary_points representing the edges of the defects contained in the picture, filling colors on the inner walls defined by the edges and the edges by utilizing colors representing the types of the defects, attenuating the filling and the edges according to a rejection rate drop_rate of 70%, covering the picture segmentation representing the points of the defects after the attenuation, and adding a conditional_map as a conditional_map;

S214, determining training window input: determining the part of the original image which is input into the network for extracting defect information according to a detection box of the defect contained in the image read in before, performing slicing operation on the original image and the conditional_map of the conditional guide image according to a training window, obtaining an image which is not scaled, overlapping the image into a six-dimensional initial feature image, and then sending the six-dimensional initial feature image into an image processing model for training;

S215, manufacturing an expansion_mask and a shrnk_mask by adopting a MASK IN MASK method: according to the track_mask representing the training area received by the image processing model, scaling the length and the width of the track_mask to manufacture an expansion_mask and a shrank_mask according to a preset ratio, wherein the expansion_mask is slightly larger than the track_mask, and the shrank_mask is slightly smaller than the track_mask;

randomly taking one of expansion_mask and spring_mask as a training area, and taking the rest content including defect foreground information as generated prompt characteristic information;

s216, selecting nonlinear attenuation during training: when the image processing model is trained, a large attenuation value is used in the small step number, and a small attenuation value is used in the opposite way;

S217, training the image processing model to generate a defect image generation model, and evaluating a training result;

training an image processing model by adopting a pure background image without defect labeling, wherein the specific process comprises the following steps of:

S221, manufacturing training images: reading in a picture, performing data enhancement on the picture excluding 'restore' and 'random_crop', converting the picture into a Tensor format, and storing the picture as a group_ truth picture;

S222, determining a generation area: randomly generating a detection box marking-box representing a generation area, and obtaining a mask representing a background area to be learned in a range defined by the detection box marking-box;

S223, preparing a condition guide chart: the background area to be learned represented by the mask is blacked on the conditional_mask of the conditional guidance graph;

S224, determining training window input: determining the part of the original image which is input into a network for extracting defect information according to a detection box of the defect contained in the image read in before, performing slicing operation on the original image and the conditional_map of the conditional guide image according to a training window, obtaining an image which is not scaled, overlapping the image into a six-dimensional initial characteristic image, and then sending the six-dimensional initial characteristic image into an image processing model for training, namely overlapping the original image of the window and the conditional_map of the conditional guide image as input;

s225, iterative training: until the specified steps are completed, evaluating the result;

S3, importing the pure background image of the defect to be generated into a defect image generation model to finish image generation of specified defect details.

2. The defect image generating method according to claim 1, wherein in step S1, the specific process includes the steps of:

S11, manufacturing training images: any picture with industrial defects and the corresponding label of the picture are read in and converted into a Tensor format, and the picture is stored as a group_ truth picture;

s12, defining a training area: according to segmentation representing defect segmentation and category representing defect types in the section corresponding to each group_ truth picture, making a condition guidance picture condition_map with the same size as the group_ truth picture, superposing the group_ truth picture and the condition guidance picture condition_map to generate a six-dimensional input picture, and then according to the section corresponding to the group_ truth picture, representing a boundary_box with unknown defects as a defined training area;

S13, training: and manufacturing a mask representing the defective area according to the marking_box, and training the area representing the defect in windowed _input to obtain a pre-training weight.

3. The method according to claim 1, wherein in step S2, the images under different scenes include images with defect labels and pure background images without defect labels.

4. The defect image generation method according to claim 1, wherein in step S3, specifying defect details includes specifying a position, a type, and a shape of a defect in the pure background image.

5. The defect image generating method according to claim 1, wherein in step S3, the specific process includes the steps of:

s31, obtaining a background target picture: reading a pure background picture which needs to generate defects, converting the pure background picture into a Tensor format, and storing the pure background picture as a background picture;

S32, obtaining a target guide diagram: taking out from the existing database, or manually designating a condition guide map, which can represent the type, position and shape detail information of the defect to be generated, and superposing the detail information with the background picture to generate an original feature map, and determining a generation_mask representing a designated generation area according to the condition guide map;

S33, obtaining window input: determining a part of the original feature map, which is input into a network for generating defect information, according to a generation_mask of a designated generation area, performing slicing operation on the original feature map according to a training window to obtain a feature map which is not scaled, and determining a mask which needs to be generated in the window;

s34, manufacturing an expansion_mask and a shrnk_mask by adopting a MASK IN MASK method: manufacturing an expansion_mask and a krink_mask according to windowed _generate_mask representing a training area received by a defect image generation model through designating a ratio;

S35, generating defect contents according to the steps: firstly reconstructing defect content in a spring_mask to obtain feature information of defect prompt in an expansion_mask range;

reconstructing defect content in the expansion_mask to obtain feature information of defect prompt in the shrnk_mask;

S36, finishing defect generation and pasting defects: after the defect feature generation is completed, the generated defect feature is pasted back into the original image through a training window, and the defect generation flow is completed.