CN117058490A - Model training method, defect image generation method and related devices - Google Patents
Model training method, defect image generation method and related devices Download PDFInfo
- Publication number
- CN117058490A CN117058490A CN202311317054.6A CN202311317054A CN117058490A CN 117058490 A CN117058490 A CN 117058490A CN 202311317054 A CN202311317054 A CN 202311317054A CN 117058490 A CN117058490 A CN 117058490A
- Authority
- CN
- China
- Prior art keywords
- model
- image
- defect
- lora
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007547 defect Effects 0.000 title claims abstract description 235
- 238000012549 training Methods 0.000 title claims abstract description 115
- 238000000034 method Methods 0.000 title claims abstract description 71
- QVFWZNCVPCJQOP-UHFFFAOYSA-N chloralodol Chemical compound CC(O)(C)CC(C)OC(O)C(Cl)(Cl)Cl QVFWZNCVPCJQOP-UHFFFAOYSA-N 0.000 claims abstract description 79
- 239000011159 matrix material Substances 0.000 claims description 26
- 238000007781 pre-processing Methods 0.000 claims description 8
- 238000013480 data collection Methods 0.000 claims description 7
- 238000002372 labelling Methods 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 6
- 239000000758 substrate Substances 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 230000002950 deficient Effects 0.000 claims description 3
- 238000001514 detection method Methods 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 7
- 238000013473 artificial intelligence Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000005336 cracking Methods 0.000 description 2
- 238000009792 diffusion process Methods 0.000 description 2
- 230000001788 irregular Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- ATJFFYVFTNAWJD-UHFFFAOYSA-N Tin Chemical compound [Sn] ATJFFYVFTNAWJD-UHFFFAOYSA-N 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 229910000679 solder Inorganic materials 0.000 description 1
- 238000005476 soldering Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
Abstract
The application provides a model training method, a defect image generation method and a related device, and relates to the field of images. Firstly, obtaining a training set, wherein the training set comprises a plurality of true defect image samples belonging to a specified defect type, the true defect image samples carry defect description labels, and then obtaining a specified image to generate a base model; finally, generating a base model based on the image, and performing LoRA training by using a training set to obtain a LoRA model conforming to the specified defect type; the LoRA model is used for carrying out local weight adjustment on the image generation base model, so that the adjusted image generation base model is used for generating a defect image conforming to a specified defect type. Because of the characteristic of extremely small sample size required by LoRA training, the process of obtaining the LoRA model through training is efficient and quick, and a defect image conforming to a specified defect type can be generated by generating a base model based on the adjusted image in the follow-up process, so that the problem of small sample of industrial defect detection is solved.
Description
Technical Field
The application relates to the field of images, in particular to a model training method, a defect image generation method and a related device.
Background
With rapid development of industry, the problem of defect detection of industrial products is more and more emphasized, when AI (Artificial Intelligence ) is mostly adopted for industrial quality inspection, a large number of defect sample pictures are needed when AI model training is performed, a large number of defect data pictures are needed, and especially some defect samples with low actual occurrence rate but serious defects can ensure the accuracy of AI industrial quality inspection, otherwise, the model training effect is poor and further the quality inspection effect is poor.
However, small samples are a common problem in the defect detection direction of many industries, and the problem is particularly obvious in the industrial detection process. At present, the following two ways are available to solve the problem of small samples in the defect detection of industrial products:
the first is engineering way, in which the first is to obtain a defect sample by manually manufacturing defects on a real product, but this is inefficient and there is a risk of irreversible damage to the product leading to wear, which is costly; the second is to manually manufacture the simulated defects based on the real images, but this approach is also inefficient and does not guarantee compliance with the real defect features;
the second is to obtain a defect sample by means of traditional OpenCV template matching, but the universality of the method is low, and high false detection rate is easy to cause.
Disclosure of Invention
The application aims to provide a model training method, a defect image generation method and a related device, so as to solve the problems in the prior art.
Embodiments of the application may be implemented as follows:
in a first aspect, the present application provides a model training method, including:
obtaining a training set, wherein the training set comprises a plurality of true defect image samples belonging to a specified defect type, and the true defect image samples carry defect description labels;
acquiring a designated image to generate a base model;
generating a base model based on the image, and performing LoRA training by using the training set to obtain a LoRA model conforming to the specified defect type; the LoRA model is used for carrying out local weight adjustment on the image generation base model, so that the adjusted image generation base model is used for generating a defect image conforming to the specified defect type.
In an alternative embodiment, the step of obtaining the training set includes:
acquiring a plurality of original defect images;
preprocessing each original defect image;
and labeling each preprocessed original defect image to obtain the training set.
In an alternative embodiment, the step of generating a base model based on the image, and performing the lore training by using the training set to obtain the lore model conforming to the specified defect type includes:
constructing an initial LoRA model; the initial LoRA model comprises a first weight matrix and a second weight matrix;
performing parameter superposition on the initial LoRA model and the image generation base model to obtain an initial image generation model;
inputting the training set into the initial image generation model, and outputting a predicted defect image corresponding to each real defect image sample under the control of the defect description label;
calculating a loss value based on each real defect image sample and the corresponding predicted defect image;
and updating the first weight matrix and the second weight matrix by using the loss value to obtain the LoRA model.
In a second aspect, the present application provides a defect image generating method, including:
obtaining a LoRA model; the LoRA model is obtained based on the model training method in the previous embodiment;
acquiring a basic image; the basic image comprises a black designated area;
loading a designated image to generate a base model;
local weight adjustment is carried out on the image generation substrate model by utilizing the LORA model, so that an image generation model is obtained;
inputting the basic image into the image generation model to obtain at least one defect image, wherein the designated area of the defect image comprises article defects conforming to designated defect types.
In an alternative embodiment, after the step of acquiring the base image, the method further includes:
acquiring a description tag conforming to the specified defect type;
the step of inputting the basic image into the image generation model to obtain at least one defect image comprises the following steps:
inputting the basic image into the image generation model to output the at least one defect image under the control of the description label.
In a third aspect, the present application provides a model training apparatus comprising:
the data collection module is used for obtaining a training set, wherein the training set comprises a plurality of true defect image samples belonging to a specified defect type, and the true defect image samples carry defect description labels;
training module for:
acquiring a designated image to generate a base model;
generating a base model based on the image, and performing LoRA training by using the training set to obtain a LoRA model conforming to the specified defect type; the LoRA model is used for carrying out local weight adjustment on the image generation base model, so that the adjusted image generation base model is used for generating a defect image conforming to the specified defect type.
In an alternative embodiment, the data collection module is specifically configured to:
acquiring a plurality of original defect images;
preprocessing each original defect image;
and labeling each preprocessed original defect image to obtain the training set.
In a fourth aspect, the present application provides a defective image generating apparatus comprising:
an acquisition module for:
obtaining a LoRA model; the LoRA model is obtained based on the model training method in the first aspect;
acquiring a basic image; the basic image comprises a black designated area;
loading a designated image to generate a base model;
a generation module for:
performing local weight adjustment on the image generation substrate model by using the LoRA model to obtain an image generation model;
inputting the basic image into the image generation model to obtain at least one defect image, wherein the designated area of the defect image comprises article defects conforming to designated defect types.
In a fifth aspect, the present application provides an electronic device, comprising: a memory storing a software program, and a processor executing the software program when the electronic device is running to implement: the model training method according to the first aspect and/or the defect image generating method according to the second aspect.
In a sixth aspect, the present application provides a computer readable storage medium storing a computer program which, when executed by a processor, implements: the model training method according to the first aspect and/or the defect image generating method according to the second aspect.
Compared with the prior art, the embodiment of the application provides a model training method, a defect image generating method and a related device, wherein a training set is firstly obtained, the training set comprises a plurality of true defect image samples belonging to a specified defect type, the true defect image samples carry defect description labels, and then a specified image generating base model is obtained; finally, generating a base model based on the image, and performing LoRA training by using a training set to obtain a LoRA model conforming to the specified defect type; the LoRA model is used for carrying out local weight adjustment on the image generation base model, so that the adjusted image generation base model is used for generating a defect image conforming to a specified defect type. Because of the characteristic of extremely small sample size required by LoRA training, the process of obtaining the LoRA model through training is efficient and quick, and a defect image conforming to a specified defect type can be generated by generating a base model based on the adjusted image in the follow-up process, so that the problem of small sample of industrial defect detection is solved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a model training method according to an embodiment of the present application.
Fig. 2 is a flowchart of a defect image generating method according to an embodiment of the present application.
Fig. 3 is a schematic diagram of image variation according to an embodiment of the present application.
Fig. 4 is a schematic structural diagram of a model training device according to an embodiment of the present application.
Fig. 5 is a schematic structural diagram of a defect image generating apparatus according to an embodiment of the present application.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. The components of the embodiments of the present application generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
In the description of the present application, it should be noted that, if the terms "upper", "lower", "inner", "outer", and the like indicate an azimuth or a positional relationship based on the azimuth or the positional relationship shown in the drawings, or the azimuth or the positional relationship in which the inventive product is conventionally put in use, it is merely for convenience of describing the present application and simplifying the description, and it is not indicated or implied that the apparatus or element referred to must have a specific azimuth, be configured and operated in a specific azimuth, and thus it should not be construed as limiting the present application.
Furthermore, the terms "first," "second," and the like, if any, are used merely for distinguishing between descriptions and not for indicating or implying a relative importance.
It should be noted that the features of the embodiments of the present application may be combined with each other without conflict.
Here, first, the keywords or key terms related to the present application will be described:
1. LoRA: the method is fully called Low-Rank Adaptation of Large Language Models, is interpreted as Low-order adaptation of a large language model, and is PEFT (parameter efficient fine tuning method). The basic principle of LoRA is to freeze pre-trained model weight parameters, and under the condition of freezing original model parameters, by adding an additional network layer into the model, only training the newly added network layer parameters. Because the number of the newly added parameters is small, the cost of finishing is obviously reduced, the effect similar to that of whole model fine tuning can be obtained, and the sample size required by LoRA training is extremely small because the parameter size required to be updated when training is required is far smaller than that required when a whole model is trained.
In the prior art, there are also ways to construct defect samples based on algorithmic approaches: for example, a neural network model represented by a countermeasure network is generated, but a great amount of training data is required for generating a defect sample by using a neural network algorithm to ensure that the generated defect sample image has higher reality, so that the neural network model is not fully applicable under the current situation that some real defect images are very few, and the generation algorithm based on the neural network often needs a great amount of time and consumes a great amount of computing resources, so that the cost is usually higher.
Based on the findings of the above technical problems, the inventors have made creative efforts to propose the following technical solutions to solve or improve the above problems. It should be noted that the above prior art solutions have all the drawbacks that the inventors have obtained after practice and careful study, and thus the discovery process of the above problems and the solutions to the problems that the embodiments of the present application hereinafter propose should not be construed as what the inventors have made in the inventive process of the present application, but should not be construed as what is known to those skilled in the art.
In view of the above, the embodiment of the application provides a model training method, which can efficiently and quickly train to obtain a LoRA model based on the characteristic that the sample size required by LoRA training is extremely small, and can generate a defect image conforming to a specified defect type by generating a base model based on an adjusted image later so as to solve the problem of small samples of industrial defect detection. The following detailed description is made by way of example with reference to the accompanying drawings.
Referring to fig. 1, fig. 1 is a flow chart of a model training method provided by an embodiment of the present application, an execution subject of the method may be an electronic device, and the training method may include steps S101 to S103 as follows:
s101, obtaining a training set.
In this embodiment, the training set includes a plurality of true defect image samples belonging to a specified defect type, each of which carries a defect description tag. Alternatively, the number of the true defect image samples in the training set depends on the requirement, for example, 5 sheets, 10 sheets, 20 sheets, etc. may be selected, which is not limited herein.
It can be understood that the defect description tag is used for describing the characteristics of the actual defects in the actual defect image sample so as to realize supervised learning.
S102, acquiring a specified image to generate a base model.
Alternatively, the image generation base model may be an AI image generation model, for example, a Stable-Diffusion model, which is a text-to-image model that generates images by inputting text, belonging to a Diffusion model in a deep learning model.
S103, generating a base model based on the image, and performing LoRA training by using the training set to obtain a LoRA model conforming to the specified defect type.
In this embodiment, the LoRA model is used to perform local weight adjustment on the image generation base model, so that the adjusted image generation base model is used to generate a defect image conforming to the specified defect type.
The model training method provided by the embodiment of the application can efficiently and quickly train to obtain the LoRA model based on the characteristic of extremely small sample size required by the LoRA training, and can generate the defect image conforming to the designated defect type by utilizing the base model generated based on the adjusted image later so as to solve the problem of small sample of industrial defect detection.
In a training set, the actual defects in each actual defect image sample belong to the same designated defect type. Taking a chip as an example, the designated defect types can be, but are not limited to, chip cracking, solder voids, etc.; taking a welded circuit board as an example, the designated defect type can be, but is not limited to, missing components, too much or too little soldering tin, and the like; taking a battery as an example, the defect type may be formulated without limitation to bulge, leakage, and the like. The above examples are merely examples, and the object to be detected by the true defect image sample is determined in actual practice, and is not limited herein.
In an alternative implementation manner, the substeps of the step S101 may include S1011 to S1013:
s1011, acquiring a plurality of original defect images.
S1012, preprocessing each original defect image.
Optionally, the preprocessing may be performed by using an image enhancement technique, where the preprocessing process may be performed on the original defect image: background removal, image enhancement, unified image size, etc.
And S1013, labeling each preprocessed original defect image to obtain a training set.
In this embodiment, the marking of each preprocessed original defect image may be a machine marking or a manual marking, which is not limited herein. And marking the preprocessed original defect image to obtain a real defect image sample carrying the defect description label.
Taking the designated defect type of the chip as chip cracking: the defect description label of a real defect image sample may be: fine straight cracks or irregular curved cracks. This example is merely an example and is not intended to be limiting herein.
In an alternative implementation manner, the substeps of the step S103 may include S1031 to S1033:
s1031, constructing an initial LoRA model.
In this embodiment, an initial LoRA model may be constructed, which may be decomposed into two low-rank matrices: a first weight matrix a and a first weight matrix and a second weight matrix B. In the initial LoRA model, a first weight matrix A and a second weight matrix B are initialized to a Gaussian distribution and a 0 matrix respectively.
The input dimension of a and the output dimension of B are the same as the input and output dimensions of the image generation substrate model W, respectively, and the output dimension of a and the input dimension of B are values far smaller than the input and output dimension of W, which is the embodiment of low-rank, so that the parameters to be trained can be greatly reduced.
S1032, carrying out parameter superposition on the initial LoRA model and the image generation base model to obtain an initial image generation model.
In this embodiment, the model weight of the image generation base model may be subjected to parameter superposition with the first weight matrix a and the second weight matrix B to obtain an initial image generation model.
S1033, inputting the training set into the initial image generation model, and outputting a predicted defect image corresponding to each real defect image sample under the control of the defect description label.
It can be understood that the defect description label is specific to the existence of a real defect in a sample of a real defect image, and the existence of a predicted defect in a predicted defect image.
S1034, calculating a loss value based on each real defect image sample and the corresponding predicted defect image.
Alternatively, the loss may be calculated based on the similarity between each true defect image sample and its corresponding predicted defect image.
S1035, updating the first weight matrix and the second weight matrix by using the loss value to obtain the LoRA model.
In this embodiment, the model weights of the original image generation base model may be regarded as "frozen", and only the first weight matrix and the second weight matrix of the lorea model need to be updated. Based on the above model training method, please refer to fig. 2, the following provides a defect image generating method, which may include the following steps S201 to S205:
s201, obtaining a LoRA model.
The LoRA model is obtained based on the model training method.
S202, acquiring a basic image.
In this embodiment, the base image includes a black designated area, and the designated area may be a regular rectangular area or an irregular curve surrounding area.
Alternatively, the base image may be obtained by superimposing a specified area mask on the basis of the normal image.
S203, loading the designated image to generate a base model.
S204, performing local weight adjustment on the image generation substrate model by using the LoRA model to obtain an image generation model.
In this embodiment, based on the idea of a weight parameter (reparameterization), the model weight of the image generation base model may be superimposed on the first weight matrix a and the second weight matrix B of the lorea model to obtain the image generation model.
S205, inputting the basic image into an image generation model to obtain at least one defect image.
In this embodiment, the designated area of the defect image may be an article defect conforming to the designated defect type.
Referring to fig. 3, fig. 3 shows a change from a normal image to a basic image to a defective image. In fig. 3, a basic image (a black block circled by a rectangular area is a designated area) is obtained after the mask image of the designated area is normally superimposed, and a defect image of an object defect conforming to the designated defect type at the designated area can be output by inputting the basic image into the image generation model.
In an alternative implementation manner, the description tag conforming to the specified defect type may be acquired while the base image is acquired, and correspondingly, the sub-step of step S205 may include:
s2051, inputting the basic image into the image generation model to output at least one defect image under the control of the descriptive label.
In an alternative implementation manner, in the case that the basic image is one, the basic image is input into the image generation model, one defect image may be output, or a plurality of defect images (the positions of the designated areas in each defect image are the same) may be input.
In another alternative implementation, N base images may be input into the image generation model, and then N defect images are output, with the designated area being the same in location in the different defect images.
It should be noted that, in the above method embodiment, the execution sequence of each step is not limited by the drawing, and the execution sequence of each step is based on the actual application situation.
According to the model training method provided by the application, the sample size required by LoRA training is very small, so that the training time required by LoRA training is greatly shortened compared with the training time of a complete deep learning large model, and the training efficiency is greatly improved.
And if a large number of defect images of various defect types are required to be generated, only a plurality of real original defect images of each defect type are required to be collected, and then a LoRA model corresponding to the defect type is obtained for each defect type based on the model training method. And for each defect type, the LoRA model corresponding to the defect type is used for combining the defect generation method, so that a defect image conforming to the defect type can be generated.
In order to perform the corresponding steps in the above method embodiments and in each possible implementation, an implementation of the model training apparatus 200 and the defect image generating apparatus 400 is given below, respectively.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a model training device according to an embodiment of the present application. The model training apparatus 200 includes: a data collection module 210 and a training module 220.
The data collection module 210 is configured to obtain a training set, where the training set includes a plurality of real defect image samples belonging to a specified defect type, and the real defect image samples carry defect description tags;
training module 220 for: acquiring a designated image to generate a base model; generating a base model based on the image, and performing LoRA training by using a training set to obtain a LoRA model conforming to the specified defect type; the LoRA model is used for carrying out local weight adjustment on the image generation base model, so that the adjusted image generation base model is used for generating a defect image conforming to a specified defect type.
Alternatively, the data collection module 210 may be specifically configured to: acquiring a plurality of original defect images; preprocessing each original defect image; labeling each preprocessed original defect image to obtain a training set.
Optionally, the training module 220 may specifically be configured to: constructing an initial LoRA model; the initial LoRA model comprises a first weight matrix and a second weight matrix; performing parameter superposition on the initial LoRA model and the image generation base model to obtain an initial image generation model; inputting the training set into an initial image generation model, and outputting a predicted defect image corresponding to each real defect image sample under the control of the defect description label; calculating a loss value based on each real defect image sample and the corresponding predicted defect image; and updating the first weight matrix and the second weight matrix by using the loss value to obtain a LoRA model.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a defect image generating apparatus according to an embodiment of the present application. The defect image generating apparatus 400 includes: an acquisition module 410 and a generation module 420.
An acquisition module 410, configured to: obtaining a LoRA model; the LoRA model is obtained based on the model training method; acquiring a basic image; the base image includes a designated area of black; loading a designated image to generate a base model;
a generating module 420, configured to: local weight adjustment is carried out on the image generation base model by using the LoRA model, so that an image generation model is obtained; inputting the basic image into an image generation model to obtain at least one defect image, wherein the designated area of the defect image comprises article defects conforming to the designated defect type.
Optionally, the obtaining module 410 may be further configured to obtain a description tag that conforms to a specified defect type; the generating module 420 may be specifically configured to input the base image into an image generating model to output at least one defect image under control of the descriptive label.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the model training apparatus 200 and/or the defect image generating apparatus 400 described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.
Referring to fig. 6, fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 300 comprises a processor 310, a memory 320 and a bus 330, the processor 310 being connected to the memory 320 via the bus 330.
The electronic device 300 may be a personal computer, a notebook computer, a server, or the like.
The memory 320 may be used to store software programs, for example, corresponding to the model training apparatus 200 and/or the defect image generating apparatus 400 as provided by the embodiments of the present application. Processor 310 performs various functional applications and data processing by running software programs stored in memory 320 to implement the model training method and/or the defect image generation method as provided by embodiments of the present application.
The Memory 320 may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), flash Memory (Flash), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
The processor 310 may be an integrated circuit chip with signal processing capabilities. The processor 310 may be a general-purpose processor including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
It is to be understood that the configuration shown in fig. 6 is illustrative only, and that electronic device 300 may also include more or fewer components than shown in fig. 6, or have a different configuration than shown in fig. 6. The components shown in fig. 6 may be implemented in hardware, software, or a combination thereof.
The embodiment of the application also provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to realize the model training method and/or the defect image generating method disclosed in the above embodiment. The computer readable storage medium may be, but is not limited to: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, RAM, PROM, EPROM, EEPROM, FLASH magnetic disk or an optical disk.
In summary, the embodiment of the application provides a model training method, a defect image generating method and a related device, wherein a training set is firstly obtained, the training set comprises a plurality of true defect image samples belonging to a specified defect type, the true defect image samples carry defect description labels, and then a specified image generation base model is obtained; finally, generating a base model based on the image, and performing LoRA training by using a training set to obtain a LoRA model conforming to the specified defect type; the LoRA model is used for carrying out local weight adjustment on the image generation base model, so that the adjusted image generation base model is used for generating a defect image conforming to a specified defect type. Because of the characteristic of extremely small sample size required by LoRA training, the process of obtaining the LoRA model through training is efficient and quick, and a defect image conforming to a specified defect type can be generated by generating a base model based on the adjusted image in the follow-up process, so that the problem of small sample of industrial defect detection is solved.
The present application is not limited to the above embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present application are intended to be included in the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (10)
1. A method of model training, comprising:
obtaining a training set, wherein the training set comprises a plurality of true defect image samples belonging to a specified defect type, and the true defect image samples carry defect description labels;
acquiring a designated image to generate a base model;
generating a base model based on the image, and performing LoRA training by using the training set to obtain a LoRA model conforming to the specified defect type; the LoRA model is used for carrying out local weight adjustment on the image generation base model, so that the adjusted image generation base model is used for generating a defect image conforming to the specified defect type.
2. The method of claim 1, wherein the step of obtaining a training set comprises:
acquiring a plurality of original defect images;
preprocessing each original defect image;
and labeling each preprocessed original defect image to obtain the training set.
3. The method of claim 1, wherein the step of generating a base model based on the image, using the training set for lore training, resulting in a lore model that conforms to the specified defect type, comprises:
constructing an initial LoRA model; the initial LoRA model comprises a first weight matrix and a second weight matrix;
performing parameter superposition on the initial LoRA model and the image generation base model to obtain an initial image generation model;
inputting the training set into the initial image generation model, and outputting a predicted defect image corresponding to each real defect image sample under the control of the defect description label;
calculating a loss value based on each real defect image sample and the corresponding predicted defect image;
and updating the first weight matrix and the second weight matrix by using the loss value to obtain the LoRA model.
4. A defect image generating method, characterized by comprising:
obtaining a LoRA model; the LoRA model is obtained based on the model training method of any one of claims 1-3;
acquiring a basic image; the basic image comprises a black designated area;
loading a designated image to generate a base model;
performing local weight adjustment on the image generation substrate model by using the LoRA model to obtain an image generation model;
inputting the basic image into the image generation model to obtain at least one defect image, wherein the designated area of the defect image comprises article defects conforming to designated defect types.
5. The method of claim 4, further comprising, after the step of acquiring the base image:
acquiring a description tag conforming to the specified defect type;
the step of inputting the basic image into the image generation model to obtain at least one defect image comprises the following steps:
inputting the basic image into the image generation model to output the at least one defect image under the control of the description label.
6. A model training device, comprising:
the data collection module is used for obtaining a training set, wherein the training set comprises a plurality of true defect image samples belonging to a specified defect type, and the true defect image samples carry defect description labels;
training module for:
acquiring a designated image to generate a base model;
generating a base model based on the image, and performing LoRA training by using the training set to obtain a LoRA model conforming to the specified defect type; the LoRA model is used for carrying out local weight adjustment on the image generation base model, so that the adjusted image generation base model is used for generating a defect image conforming to the specified defect type.
7. The apparatus of claim 6, wherein the data collection module is specifically configured to:
acquiring a plurality of original defect images;
preprocessing each original defect image;
and labeling each preprocessed original defect image to obtain the training set.
8. A defective image generating apparatus, comprising:
an acquisition module for:
obtaining a LoRA model; the LoRA model is obtained based on the model training method of any one of claims 1-3;
acquiring a basic image; the basic image comprises a black designated area;
loading a designated image to generate a base model;
a generation module for:
performing local weight adjustment on the image generation substrate model by using the LoRA model to obtain an image generation model;
inputting the basic image into the image generation model to obtain at least one defect image, wherein the designated area of the defect image comprises article defects conforming to designated defect types.
9. An electronic device, comprising: a memory storing a software program, and a processor executing the software program when the electronic device is running to implement: a model training method according to any one of claims 1-3, and/or a defect image generation method according to any one of claims 4-5.
10. A computer readable storage medium, wherein the computer readable storage medium stores a computer program which when executed by a processor implements: a model training method according to any one of claims 1-3, and/or a defect image generation method according to any one of claims 4-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311317054.6A CN117058490A (en) | 2023-10-12 | 2023-10-12 | Model training method, defect image generation method and related devices |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311317054.6A CN117058490A (en) | 2023-10-12 | 2023-10-12 | Model training method, defect image generation method and related devices |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117058490A true CN117058490A (en) | 2023-11-14 |
Family
ID=88664855
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311317054.6A Pending CN117058490A (en) | 2023-10-12 | 2023-10-12 | Model training method, defect image generation method and related devices |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117058490A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113264200A (en) * | 2021-04-14 | 2021-08-17 | 泛博科技(中山)有限公司 | Aviation intelligent self-checking multi-scene universal unmanned vehicle system |
| CN116524299A (en) * | 2023-05-04 | 2023-08-01 | 中国兵器装备集团自动化研究所有限公司 | Image sample generation method, device, equipment and storage medium |
| CN116630464A (en) * | 2023-07-21 | 2023-08-22 | 北京蔚领时代科技有限公司 | Image style migration method and device based on stable diffusion |
| CN116824005A (en) * | 2023-06-27 | 2023-09-29 | Oppo广东移动通信有限公司 | Image processing method and device, storage medium and electronic equipment |
| CN116824004A (en) * | 2023-06-27 | 2023-09-29 | Oppo广东移动通信有限公司 | Icon generation method and device, storage medium and electronic equipment |
| CN116862902A (en) * | 2023-07-31 | 2023-10-10 | 厦门微图软件科技有限公司 | Method for generating defects based on DreamBooth fine-tuning Stable Diffusion model |
-
2023
- 2023-10-12 CN CN202311317054.6A patent/CN117058490A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113264200A (en) * | 2021-04-14 | 2021-08-17 | 泛博科技(中山)有限公司 | Aviation intelligent self-checking multi-scene universal unmanned vehicle system |
| CN116524299A (en) * | 2023-05-04 | 2023-08-01 | 中国兵器装备集团自动化研究所有限公司 | Image sample generation method, device, equipment and storage medium |
| CN116824005A (en) * | 2023-06-27 | 2023-09-29 | Oppo广东移动通信有限公司 | Image processing method and device, storage medium and electronic equipment |
| CN116824004A (en) * | 2023-06-27 | 2023-09-29 | Oppo广东移动通信有限公司 | Icon generation method and device, storage medium and electronic equipment |
| CN116630464A (en) * | 2023-07-21 | 2023-08-22 | 北京蔚领时代科技有限公司 | Image style migration method and device based on stable diffusion |
| CN116862902A (en) * | 2023-07-31 | 2023-10-10 | 厦门微图软件科技有限公司 | Method for generating defects based on DreamBooth fine-tuning Stable Diffusion model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109783635A (en) | Use machine learning and fuzzy matching AUTOMATIC ZONING classifying documents and identification metadata | |
| CN110968654B (en) | Address category determining method, equipment and system for text data | |
| Lu et al. | Defect detection of PCB based on Bayes feature fusion | |
| Zhao et al. | Real‐time fabric defect detection based on multi‐scale convolutional neural network | |
| CN111950279A (en) | Entity relationship processing method, device, equipment and computer readable storage medium | |
| CN115795000A (en) | Joint similarity algorithm comparison-based enclosure identification method and device | |
| CN111582309B (en) | Method for generating bad point detection model of design layout and method for detecting bad point | |
| CN114387289B (en) | Semantic segmentation method and device for three-dimensional point cloud of power transmission and distribution overhead line | |
| CN111401309A (en) | CNN training and remote sensing image target identification method based on wavelet transformation | |
| CN112861934A (en) | Image classification method and device of embedded terminal and embedded terminal | |
| CN117058490A (en) | Model training method, defect image generation method and related devices | |
| CN115049836B (en) | Image segmentation method, device, equipment and storage medium | |
| Zhang et al. | A spatial–spectral adaptive learning model for textile defect images recognition with few labeled data | |
| CN115359203A (en) | Three-dimensional high-precision map generation method and system and cloud platform | |
| CN115205619A (en) | Training method, detection method, device and storage medium for detection model | |
| Dorrer et al. | Normalization of data for training and analysis by the MaskRCNN model using the k-means method for a smart refrigerator’s computer vision | |
| CN115116080A (en) | Table analysis method and device, electronic equipment and storage medium | |
| CN113780239A (en) | Iris recognition method, iris recognition device, electronic equipment and computer readable medium | |
| CN112966038A (en) | Method and device for extracting structured data from unstructured data | |
| CN112084364A (en) | Object analysis method, local image search method, device, and storage medium | |
| Fahmy | A new Zernike moments based technique for camera identification and forgery detection | |
| Singh et al. | Application of error level analysis in image spam classification using deep learning model | |
| CN115563289B (en) | Industry classification label generation method and device, electronic equipment and readable medium | |
| CN116341531B (en) | Knowledge-driven character information extraction and inspection method and device | |
| CN117612181A (en) | Image recognition method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |