CN114049536A - Virtual sample generation method and device, storage medium and electronic equipment - Google Patents
Virtual sample generation method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN114049536A CN114049536A CN202111365555.2A CN202111365555A CN114049536A CN 114049536 A CN114049536 A CN 114049536A CN 202111365555 A CN202111365555 A CN 202111365555A CN 114049536 A CN114049536 A CN 114049536A
- Authority
- CN
- China
- Prior art keywords
- sample
- data
- samples
- virtual
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000013508 migration Methods 0.000 claims abstract description 21
- 230000005012 migration Effects 0.000 claims abstract description 21
- 230000003042 antagnostic effect Effects 0.000 claims abstract description 6
- 235000019504 cigarettes Nutrition 0.000 claims description 94
- 230000015654 memory Effects 0.000 claims description 24
- 230000003321 amplification Effects 0.000 claims description 11
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 11
- 238000004422 calculation algorithm Methods 0.000 claims description 10
- 239000002775 capsule Substances 0.000 claims description 8
- 238000007418 data mining Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 5
- 230000008485 antagonism Effects 0.000 claims description 3
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 12
- 238000012549 training Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 10
- 230000001965 increasing effect Effects 0.000 description 10
- 230000000694 effects Effects 0.000 description 7
- 238000012856 packing Methods 0.000 description 6
- 230000003190 augmentative effect Effects 0.000 description 5
- 238000001514 detection method Methods 0.000 description 5
- 238000013507 mapping Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000003707 image sharpening Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a virtual sample generation method, a virtual sample generation device, a storage medium and electronic equipment, wherein the method comprises the following steps: acquiring a background scene sample and a data sample to be detected; constructing a sample relation model according to the information of the data sample; performing data enhancement on a data sample to be detected to obtain an enhanced data sample; arranging the enhanced data samples according to an arrangement rule constructed by a sample relation model, and embedding the enhanced data samples into background scene samples to generate first samples; and carrying out style migration on the first sample according to the antagonistic network to obtain a virtual sample. By implementing the method, in the process of generating the high-quality virtual sample, the sample arrangement in the generated virtual sample is more consistent with the actual application scene by introducing the relational modeling, and the data sample is subjected to style migration by utilizing the generation countermeasure network, so that the finally generated sample style has consistency, and the sample image can be smoothly embedded into the background, so that the image is more vivid.
Description
Technical Field
The invention relates to the technical field of computer vision and pattern recognition, in particular to a virtual sample generation method and device, a storage medium and electronic equipment.
Background
In recent years, the artificial intelligence technology is rapidly developed in the industry, and people use a deep learning method to perform image recognition and target detection, so that the people are helped to fulfill various requirements. The classification and detection model constructed by the neural network algorithm has better effect in industry. The image recognition technology and other technologies need to rely on the drive of big data, and can achieve better practical application effect after training and learning the model by a large amount of training data.
In an actual application scenario, data is often insufficient, and a model cannot learn to obtain a good expression under a small amount of data, so that in order to obtain a better algorithm effect, more pictures containing rich features need to be obtained by a manual means. However, the larger the amount of sample, not only the cost is increased, but also the difficulty of obtaining the sample is increased.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, a storage medium, and an electronic device for generating a virtual sample, so as to solve the technical problem in the prior art that training samples are few when a model is trained and learned.
The technical scheme provided by the invention is as follows:
a first aspect of an embodiment of the present invention provides a method for generating a virtual sample, including: acquiring a background scene sample and a data sample to be detected; constructing a sample relation model according to the information of the data sample; performing data enhancement on a data sample to be detected to obtain an enhanced data sample; arranging the enhanced data samples according to an arrangement rule constructed by the sample relation model, and then embedding the enhanced data samples into the background scene samples to generate first samples; and carrying out style migration on the first sample according to the antagonistic network to obtain a virtual sample.
Optionally, the data enhancement mode includes: random dithering of scales, data enhancement based on HSV space, image sharpness enhancement, and random flipping and rotation; carrying out data enhancement on a data sample to be detected, comprising the following steps: and performing data enhancement on the data samples to be detected randomly according to the preset probability for each data amplification mode.
Optionally, constructing a sample relationship model according to the information of the data sample includes: acquiring sales information of the data sample; determining the specification information of the data sample according to the sales information; based on a data mining algorithm, calculating the relevance of the brand sales volume of each data sample and the sales volumes of other brands according to the sales information and the product specification information to obtain a data product specification with the relevance larger than a threshold value; and constructing a sample relation model according to the specification information of the data sample and the data specification of which the correlation is greater than the threshold value.
Optionally, the arranging the enhanced data samples according to the arrangement rule constructed by the sample relationship model and then embedding the enhanced data samples into the background scene samples to generate a first sample, including: arranging the data samples according to a random placement mode, specification information or correlation information to obtain the arranged data samples; and embedding the arranged data samples into the background scene sample to generate a first sample.
Optionally, the loss function of the antagonistic network is represented by the following formula:
wherein,
representing antagonism loss, G representing the generator, DYThe presence of the discriminator is indicated by the expression,
indicating a loss of cyclic consistency.
Optionally, performing style migration on the first sample according to the antagonistic network to obtain a virtual sample, including: cutting each data sample image in the first sample, inputting the cut data sample image into a countermeasure network for style migration, and obtaining a migrated data sample image; and embedding the transferred data sample image into the first sample to obtain a virtual sample.
Optionally, the data sample to be detected is a cigarette packet outer package sample.
A second aspect of the embodiments of the present invention provides a virtual sample generation apparatus, including: the sample acquisition module is used for acquiring a background scene sample and a data sample to be detected; the relational model building module is used for building a sample relational model according to the information of the data samples; the enhancement module is used for enhancing the data of the data sample to be detected to obtain an enhanced data sample; the arrangement module is used for arranging the enhanced data samples according to the arrangement rule established by the sample relation model and then embedding the enhanced data samples into the background scene samples to generate first samples; and the style migration module is used for carrying out style migration on the first sample according to the confrontation type network to obtain a virtual sample.
A third aspect of the embodiments of the present invention provides a computer-readable storage medium, where computer instructions are stored, and the computer instructions are configured to cause a computer to execute the virtual sample generation method according to any one of the first aspect and the first aspect of the embodiments of the present invention.
A fourth aspect of an embodiment of the present invention provides an electronic device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the virtual sample generation method according to any one of the first aspect and the first aspect of the embodiments of the present invention.
The technical scheme provided by the invention has the following effects:
according to the virtual sample generation method, the virtual sample generation device, the storage medium and the electronic equipment provided by the embodiment of the invention, in the process of generating the high-quality virtual sample, the sample arrangement in the generated virtual sample is enabled to be more consistent with the actual application scene by introducing the relational modeling, and meanwhile, the data sample is subjected to style migration by utilizing the generation impedance network, so that the style of the finally generated sample has consistency, and the sample image can be smoothly embedded into the background, so that the image is enabled to be more vivid.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow diagram of a virtual sample generation method according to an embodiment of the invention;
FIG. 2 is a schematic diagram of a data enhancement method based on image size random dithering according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an HSV space-based data enhancement according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a data enhancement mode based on image sharpness enhancement according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data enhancement method based on random flipping and rotation according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a virtual sample generated by a virtual sample generation method according to an embodiment of the invention;
FIG. 7 is a flow diagram of a virtual sample generation method according to another embodiment of the invention;
fig. 8 is a block diagram of a configuration of a virtual sample generation apparatus according to an embodiment of the present invention;
FIG. 9 is a schematic structural diagram of a computer-readable storage medium provided in accordance with an embodiment of the present invention;
fig. 10 is a schematic structural diagram of an electronic device provided in an embodiment of the present invention.
Detailed Description
As described in the background art, the model cannot learn a good performance under a small amount of data, and therefore, in order to obtain a better algorithm effect, it is necessary to obtain more pictures containing rich features by a manual means. At present, two ways of increasing samples are adopted, one way is to increase the times of training samples by adjusting the sampling frequency, but the method may cause overfitting of the model to the training samples, and simultaneously, the robustness of the model is also reduced, and the actual performance of the model may be reduced instead. The other method is to perform data enhancement on a small number of data samples, and to perform minor adjustments such as rotation, displacement, inversion and the like by using the existing data set images, which introduces a priori knowledge semantic invariance, that is, the operations do not change the semantic information of the images, and through the operations, the network can be made more robust, and at the same time, the images of the categories with a small number are supplemented.
In the field of target detection, one image contains samples to be identified of different types, and meanwhile, the position information of the target needs to be positioned, namely the quality of the generated sample is poor when the sample is increased by adopting the second mode. Therefore, a more complicated process is also required when the sample size is increased.
In view of this, an embodiment of the present invention provides a virtual sample generation method, including: acquiring a background scene sample and a data sample to be detected; constructing a sample relation model according to the specification information of the data sample; performing data enhancement on a data sample to be detected to obtain an enhanced data sample; arranging the enhanced data samples according to an arrangement rule constructed by the sample relation model to obtain arranged data samples; embedding the arranged data samples into the background scene sample to generate a first sample; cutting each data sample image in the first sample, inputting the cut data sample image into a countermeasure network for style migration, and obtaining a migrated data sample image; and embedding the transferred data sample image into the first sample to obtain a virtual sample.
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
An embodiment of the present invention provides a virtual sample generation method, as shown in fig. 1, the method includes the following steps:
step S101: and acquiring a background scene sample and a data sample to be detected.
In particular, the data sample may be data that requires an increased sample. In one embodiment, the data is a sample of a cigarette capsule overwrap. At present, when a neural network is adopted to detect cigarette images, images of cigarette retail terminals are generally obtained. However, due to the fact that the shelf loading rate of part of cigarette specifications (brands and specifications) is low, the number of the collected images is small, so that the categories cannot be trained sufficiently during model training, and the detection effect of the models is influenced. Therefore, the virtual sample can be used for generating the virtual sample for the cigarette small box outer package image, and the sample amount can be increased. The data sample may be other data that requires an increased sample.
In one embodiment, when the data sample to be detected is a cigarette capsule overwrap sample, the background scene sample may be a sample containing an image of a retail cabinet in which the cigarettes are placed.
Step S102: and constructing a sample relation model according to the information of the data samples. Specifically, in order to make the generated virtual sample more fit to the actual application scenario and improve the quality of the generated virtual sample, a sample relationship model may be constructed according to the relevant information of the data sample. For example, in order to make the cigarette virtual sample fit a retail scene better, the retail or sales information of the cigarette can be obtained to construct a relational model of the cigarette.
In one embodiment, constructing a sample relationship model from information of data samples includes: acquiring sales information of the data sample; determining the specification information of the data sample according to the sales information; based on a data mining algorithm, calculating the relevance of the brand sales volume of each data sample and the sales volumes of other brands according to the sales information and the product specification information to obtain a data product specification with the relevance larger than a threshold value; and constructing a sample relation model according to the specification information of the data sample and the data specification of which the correlation is greater than the threshold value.
For a cigarette sample, the information of the cigarette specifications sold in a retail store and the related data of cigarette sales can be collected, the relevance between the sales volume of each cigarette brand and the sales volume of other cigarette brands can be calculated according to the collected cigarette sales information by using a data mining algorithm such as an Apriori algorithm, 5 other cigarette specifications with the maximum relevance to each cigarette specification in the cigarette sample can be obtained after calculation by inputting the cigarette brands needing to be calculated into a data mining algorithm model, and the other cigarette specifications and the information of the cigarette specifications are stored in a database together to obtain a cigarette relation model. Therefore, several cigarette brand specifications with highest relevance to the specification of a specified certain cigarette brand can be obtained by inquiring the database, and information such as a cigarette brand, a manufacturer and the like to which the cigarette with the certain cigarette brand specification belongs can also be obtained by inquiring the database.
Step S103: and performing data enhancement on the data sample to be detected to obtain the enhanced data sample. Specifically, because a large number of high-quality virtual samples need to be generated by a small amount of image data, the samples generated by using a simple map cutting and mapping method cannot meet the requirement of large batch of cigarette packet outer package image samples, and meanwhile, the diversity of the images is also lacked. In order to enhance the diversity of generated images and improve the quality of amplification data, a random data enhancement means for a sample is adopted in each data amplification process, so that the richness of the images is increased. Therefore, when the data sample to be detected is subjected to data enhancement, a plurality of data amplification modes can be selected, and the data sample to be detected is subjected to data enhancement randomly according to a preset probability aiming at each data amplification mode.
In one embodiment, for a cigarette sample, the data amplification strategy is performed primarily around both the color channel and the geometric transformation. Thus, the data enhancement mode comprises the following steps: the image scale is subjected to random dithering, data enhancement based on an HSV color space, image sharpness enhancement, random turning, rotation and other enhancement means, so that the richness of the virtual sample in generation is increased, and images generated every time are different.
As shown in fig. 2, the random scale dithering is to select a coefficient between 0.5 and 1.5 to randomly scale the size of the image to be amplified, and the size of the image after amplification is the length and width of the original image multiplied by the coefficient; as shown in fig. 3, the HSV-space-based data enhancement is to convert an image from an RGB space to an HSV space, then fine-tune the image from three dimensional values of hue, saturation and brightness, the adjustment range is maintained within a range of 0.8-1.2, image acquisition under different illumination is simulated, adjustment is performed within a small range in order to not generate an excessively distorted image, and thus the diversity of an augmented image is further enhanced, and the RGB-to-HSV conversion formula is represented by the following formula:
H＝0°,max(R,G,B)＝min(R,G,B)
as shown in fig. 4, image sharpness enhancement is to adjust the edge characteristics and overall sharpness of an image by image sharpening; as shown in fig. 5, the random flipping and rotation is to randomly perform horizontal flipping operation and small-angle oblique rotation on the augmented cigarette example, so as to simulate multiple placing modes of cigarettes during placing, and enhance the robustness of the model.
Based on the four data amplification methods mentioned above, ten methods for amplifying a sample can be obtained. The method specifically comprises the following steps: 1. horizontal movement of the bounding box 2, vertical movement of the bounding box 3, size dithering of the bounding box 4, brightness adjustment of the bounding box 5, saturation adjustment of the bounding box 6, hue adjustment of the bounding box 7, sharpening adjustment of the bounding box 8, horizontal flipping of the bounding box 9, rotation of the bounding box 10, histogram equalization.
In data enhancement, the ten data amplification methods mentioned above can be adopted to randomly amplify the image based on a preset probability. For example, the predetermined probability is 0.5, then, by ten data amplification methods, theoretically each image can be amplified to obtain 2101024 different samples. However, in practical applications, the cigarette data sample is usually obtained by cutting the cigarette container image collected in a shop, and when the cigarette sample is obtained by cutting, not only the cigarette image but also the background information around the cigarette is cut. Thus, even if the cigarette images are the same, the background information around the cigarette images is different (for example, the containers used are different). In addition, when the cut cigarette samples are actually processed, dozens of cut cigarette samples are also processed in a map modeDifferent cigarette samples are attached to one image. Thus, even for the same cigarette image, the background information of the surroundings and the arrangement of the cigarettes around are different from each other.
Therefore, when data enhancement is performed on each cigarette sample in one image, the position information (including the surrounding background information and the surrounding cigarette arrangement) around each cigarette sample is different. Therefore, compared with a mode of directly performing oversampling and the like, the data enhancement by the mode can prevent the network overfitting and enhance the robustness of the network. Meanwhile, through data enhancement, a great amount of external packing images of the cigarette small boxes are adopted, a great amount of external packing samples of the virtual cigarette small boxes are generated, and a training data set for balancing external packing identification of the cigarette small boxes can be expanded, so that the purposes of enriching data diversity and cigarette category balance are achieved.
Step S104: arranging the enhanced data samples according to the arrangement rule established by the sample relation model, and then embedding the enhanced data samples into the background scene samples to generate a first sample. In particular, although the enhanced data samples satisfy the requirement of the augmented samples, the quality of the resulting augmented samples or augmented training data set cannot satisfy the requirement. Therefore, in order to enable the enhanced data samples to better conform to the actual situation, the enhanced data samples can be arranged by adopting an arrangement rule constructed based on the sample relation model.
In one embodiment, the arrangement rule includes a random placement manner, and a manner of arranging the data samples based on the specification information or the correlation information. The random placement mode is that cigarette samples of different types are completely randomly selected, different positions are randomly selected for placement, and when the cigarette samples are placed, the positions and the sizes of the cigarette samples need to be paid attention to and do not exceed the limit. The placement based on the cigarette gauge information means that one cigarette gauge is randomly selected when each row is placed, and the cigarette gauges of all manufacturers are selected to be placed after the relational model is input, so that the cigarette gauges of each row are the same. The placement based on the correlation information means that for certain types of cigarette samples, a plurality of cigarette product gauges which are most related to the cigarette product gauges are obtained through a relation model, and the product gauges are selected to be placed on the left side and the right side of the cigarette product gauges.
After the enhanced sample is placed in the placing mode, the enhanced sample can be embedded into the background scene sample to obtain a first sample. In particular, the laid sample may be embedded in one of the background scene samples or in a retail cabinet of the background scene sample. Thus, the limit when laying out may be determined based on the background scene sample. The enhanced data samples are arranged according to the arrangement rule established by the sample relation model, so that various cigarette position placing modes in a real retail scene can be simulated, the formed first sample is closer to a real scene, and the quality is higher.
Step S105: and carrying out style migration on the first sample according to the antagonistic network to obtain a virtual sample.
When the images are collected, the illumination, the visual angle and the like of each image are different when the images are shot, so the image style of each image is different, the styles of different cigarette product images are possibly not uniform in the process of generating the virtual sample, the generated virtual sample and the real collected image have larger difference, and simultaneously, when the enhanced cigarette image is directly embedded into the background image, the edge of the cigarette image and the original background image have larger difference and are not smooth enough, and the quality of the generated virtual sample is also influenced. In order to solve the problem, the style of the image is transferred by using a countermeasure network (CycleGAN), so that the image keeps a relatively consistent style.
Before the countermeasure network is adopted for style migration, the real cigarette display image and the artificially generated chartlet image are used for training the cycleGAN network to obtain the trained countermeasure network. The loss function of the network is as follows:
wherein,
representing a loss of antagonism, a forged sample G (X) is obtained by the generator G inputting samples in domain X, with the aim of making G (X) as similar as possible to the sample Y in domain Y, and a discriminator DYThe goal of (a) is to distinguish y from G (X) as much as possible.
Indicating the loss of cyclic consistency, and is used for restricting the sample G (X) obtained by the generator to be consistent with the original sample x in content.
Therefore, when the style is transferred, the trained cycleGAN network is used for cutting each cigarette capsule outer package sample in the virtual sample and the surrounding context area thereof, the cut cigarette capsule outer package sample is sent into the cycleGAN network, the image after the style transfer is output by the network, and the overall style of the cigarette capsule outer package sample is consistent with that of the background image through the image. The data sample image after the style migration can be embedded into the background scene sample of the first sample, so that a high-quality virtual sample is obtained. As shown in fig. 6, the result of the virtual sample generated by the virtual sample generation method is shown.
According to the virtual sample generation method provided by the embodiment of the invention, in the process of generating the high-quality virtual sample, the sample arrangement in the generated virtual sample is more consistent with the actual application scene by introducing the relational modeling, and meanwhile, the data sample is subjected to style migration by utilizing the generation countermeasure network, so that the style of the finally generated sample is consistent, and the sample image can be smoothly embedded into the background, so that the image is more vivid.
In one embodiment, the virtual sample generation method can be applied to the virtual sample generation process of the cigarette small box outer package image; as shown in fig. 7, the generation process is implemented by the following flow: the method comprises the steps of utilizing a small amount of cigarette small box outer packing images with rare specifications as a basis for virtual sample generation, generating enhanced training pictures according to the arrangement rule of retail cigarette images and the characteristic of scene fixation by using collected pictures of rare cigarette types through a mapping means and a data enhancement method of the images, adding the enhanced training pictures into a data centralized training model, thereby improving the recognition capability of the cigarette specifications with rare images, constructing a cigarette type relation model for ensuring high quality of artificially generated virtual samples, modeling different sold cigarette types, analyzing the relation among the cigarette types during cigarette display, and utilizing the rule to match the cigarette arrangement when the virtual samples are generated. In order to eliminate style inconsistency and image edge difference generated in the process of mapping, a style transfer conversion mode is learned by a cycleGAN algorithm, and a style conversion module is used for generating a cigarette outer package sample image which is more practical.
The virtual sample generation method provided by the embodiment of the invention aims at the problems that the quantity of the external packing samples of certain cigarette small boxes provided in the image acquisition of the retail counter is small and the training is insufficient, and artificially generates the external packing virtual samples of the cigarette small boxes by using the collected small box pictures of rare types of cigarettes and using a mapping means. Specifically, the characteristics that the retail cigarette small box images are regular in arrangement and relatively fixed in scene are utilized, a large number of high-quality virtual samples are artificially generated through a small number of rare types of cigarette small box outer package image samples and various data enhancement means. Meanwhile, in order to generate a virtual sample more real, a more targeted cigarette position placing strategy is generated by utilizing the relevance among different cigarette categories, then the style of the background image of the cigarette small box outer package in each virtual sample is unified by utilizing a cycleGAN network, and the generated sample is more vivid due to the smoothly embedded cigarette image boundary and more accords with the cigarette small box outer package image in a real retail scene. Therefore, the method can provide a high-quality method for expanding the capsule external package training sample for deep learning methods based on data driving, such as image recognition and target detection of the cigarette capsule.
An embodiment of the present invention further provides a virtual sample generation apparatus, as shown in fig. 8, the apparatus includes:
the sample acquisition module is used for acquiring a background scene sample and a data sample to be detected; for details, reference is made to the corresponding parts of the above method embodiments, which are not described herein again.
The relational model building module is used for building a sample relational model according to the quality and specification information of the data samples; for details, reference is made to the corresponding parts of the above method embodiments, which are not described herein again.
The enhancement module is used for enhancing the data of the data sample to be detected to obtain an enhanced data sample; for details, reference is made to the corresponding parts of the above method embodiments, which are not described herein again.
The arrangement module is used for arranging the enhanced data samples according to the arrangement rule established by the sample relation model and then embedding the enhanced data samples into the background scene samples to generate first samples; for details, reference is made to the corresponding parts of the above method embodiments, which are not described herein again.
The style migration module is used for carrying out style migration on the first sample according to the countermeasure network to obtain a virtual sample; for details, reference is made to the corresponding parts of the above method embodiments, which are not described herein again.
The virtual sample generation device provided by the embodiment of the invention enables the sample arrangement in the generated virtual sample to be more consistent with the actual application scene by introducing the relational modeling in the process of generating the high-quality virtual sample, and meanwhile, the generated impedance network is utilized to carry out style migration on the data sample, so that the style of the finally generated sample has consistency, and the sample image can be smoothly embedded into the background, so that the image is more vivid.
The function description of the virtual sample generation device provided by the embodiment of the invention refers to the description of the virtual sample generation method in the above embodiment in detail.
An embodiment of the present invention further provides a storage medium, as shown in fig. 9, on which a computer program 601 is stored, where the instructions, when executed by a processor, implement the steps of the virtual sample generation method in the foregoing embodiments. The storage medium is also stored with audio and video stream data, characteristic frame data, an interactive request signaling, encrypted data, preset data size and the like. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD) or a Solid State Drive (SSD), etc.; the storage medium may also comprise a combination of memories of the kind described above.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD) or a Solid State Drive (SSD), etc.; the storage medium may also comprise a combination of memories of the kind described above.
An embodiment of the present invention further provides an electronic device, as shown in fig. 10, the electronic device may include a processor 51 and a memory 52, where the processor 51 and the memory 52 may be connected by a bus or in another manner, and fig. 10 takes the example of connection by a bus as an example.
The processor 51 may be a Central Processing Unit (CPU). The Processor 51 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or combinations thereof.
The memory 52, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the corresponding program instructions/modules in the embodiments of the present invention. The processor 51 executes various functional applications and data processing of the processor by running non-transitory software programs, instructions and modules stored in the memory 52, that is, implements the virtual sample generation method in the above method embodiment.
The memory 52 may include a storage program area and a storage data area, wherein the storage program area may store an operating device, an application program required for at least one function; the storage data area may store data created by the processor 51, and the like. Further, the memory 52 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 52 may optionally include memory located remotely from the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 52 and, when executed by the processor 51, perform the virtual sample generation method in the embodiment shown in fig. 1-7.
The details of the electronic device may be understood by referring to the corresponding descriptions and effects in the embodiments shown in fig. 1 to fig. 7, which are not described herein again.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A method for generating a virtual sample, comprising:
acquiring a background scene sample and a data sample to be detected;
constructing a sample relation model according to the information of the data sample;
performing data enhancement on a data sample to be detected to obtain an enhanced data sample;
arranging the enhanced data samples according to an arrangement rule constructed by the sample relation model, and then embedding the enhanced data samples into the background scene samples to generate first samples;
and carrying out style migration on the first sample according to the antagonistic network to obtain a virtual sample.
2. The virtual sample generation method of claim 1, wherein the data enhancement mode comprises: random dithering of scales, data enhancement based on HSV space, image sharpness enhancement, and random flipping and rotation;
carrying out data enhancement on a data sample to be detected, comprising the following steps:
and performing data enhancement on the data samples to be detected randomly according to the preset probability for each data amplification mode.
3. The virtual sample generation method of claim 1, wherein constructing a sample relationship model from information of data samples comprises:
acquiring sales information of the data sample;
determining the specification information of the data sample according to the sales information;
based on a data mining algorithm, calculating the relevance of the brand sales volume of each data sample and the sales volumes of other brands according to the sales information and the product specification information to obtain a data product specification with the relevance larger than a threshold value;
and constructing a sample relation model according to the specification information of the data sample and the data specification of which the correlation is greater than the threshold value.
4. The virtual sample generation method according to claim 1, wherein the arranging the enhanced data samples according to the arrangement rule constructed by the sample relationship model and then embedding the enhanced data samples into the background scene samples to generate the first sample includes:
arranging the data samples according to a random placement mode, specification information or correlation information to obtain the arranged data samples;
and embedding the arranged data samples into the background scene sample to generate a first sample.
5. The virtual sample generation method of claim 1, wherein the loss function of the impedance network is represented by the following formula:
wherein,
representing antagonism loss, G representing the generator, DYThe presence of the discriminator is indicated by the expression,
indicating a loss of cyclic consistency.
6. The method of claim 1, wherein performing style migration on the first sample according to the countermeasure network to obtain the virtual sample comprises:
cutting each data sample image in the first sample, inputting the cut data sample image into a countermeasure network for style migration, and obtaining a migrated data sample image;
and embedding the transferred data sample image into the first sample to obtain a virtual sample.
7. The virtual sample generation method of claim 1, wherein the data sample to be detected is a cigarette capsule overwrap sample.
8. A virtual sample generation apparatus, comprising:
the sample acquisition module is used for acquiring a background scene sample and a data sample to be detected;
the relational model building module is used for building a sample relational model according to the information of the data samples;
the enhancement module is used for enhancing the data of the data sample to be detected to obtain an enhanced data sample;
the arrangement module is used for arranging the enhanced data samples according to the arrangement rule established by the sample relation model and then embedding the enhanced data samples into the background scene samples to generate first samples;
and the style migration module is used for carrying out style migration on the first sample according to the confrontation type network to obtain a virtual sample.
9. A computer-readable storage medium storing computer instructions for causing a computer to perform the virtual sample generation method of any one of claims 1 to 7.
10. An electronic device, comprising: a memory and a processor communicatively coupled to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the virtual sample generation method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111365555.2A CN114049536A (en) | 2021-11-17 | 2021-11-17 | Virtual sample generation method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111365555.2A CN114049536A (en) | 2021-11-17 | 2021-11-17 | Virtual sample generation method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114049536A true CN114049536A (en) | 2022-02-15 |
Family
ID=80209949
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111365555.2A Pending CN114049536A (en) | 2021-11-17 | 2021-11-17 | Virtual sample generation method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114049536A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114882937A (en) * | 2022-04-30 | 2022-08-09 | 苏州浪潮智能科技有限公司 | Solid state disk durability testing method, sample amount calculating method and device |
| CN115205432A (en) * | 2022-09-03 | 2022-10-18 | 深圳爱莫科技有限公司 | Simulation method and model for automatic generation of cigarette terminal display sample image |
| CN115601631A (en) * | 2022-12-15 | 2023-01-13 | 深圳爱莫科技有限公司（Cn） | Cigarette display image recognition method, model, equipment and storage medium |
| CN116341561A (en) * | 2023-03-27 | 2023-06-27 | 京东科技信息技术有限公司 | Voice sample data generation method, device, equipment and storage medium |
-
2021
- 2021-11-17 CN CN202111365555.2A patent/CN114049536A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114882937A (en) * | 2022-04-30 | 2022-08-09 | 苏州浪潮智能科技有限公司 | Solid state disk durability testing method, sample amount calculating method and device |
| CN115205432A (en) * | 2022-09-03 | 2022-10-18 | 深圳爱莫科技有限公司 | Simulation method and model for automatic generation of cigarette terminal display sample image |
| CN115205432B (en) * | 2022-09-03 | 2022-11-29 | 深圳爱莫科技有限公司 | Simulation method and model for automatic generation of cigarette terminal display sample image |
| CN115601631A (en) * | 2022-12-15 | 2023-01-13 | 深圳爱莫科技有限公司（Cn） | Cigarette display image recognition method, model, equipment and storage medium |
| CN116341561A (en) * | 2023-03-27 | 2023-06-27 | 京东科技信息技术有限公司 | Voice sample data generation method, device, equipment and storage medium |
| CN116341561B (en) * | 2023-03-27 | 2024-02-02 | 京东科技信息技术有限公司 | Voice sample data generation method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114049536A (en) | Virtual sample generation method and device, storage medium and electronic equipment | |
| CN109583483B (en) | Target detection method and system based on convolutional neural network | |
| CN109886997B (en) | Identification frame determining method and device based on target detection and terminal equipment | |
| CN108229479B (en) | Training method and device of semantic segmentation model, electronic equipment and storage medium | |
| CN107562805B (en) | Method and device for searching picture by picture | |
| CN111161349B (en) | Object posture estimation method, device and equipment | |
| CN108121997A (en) | Use the object classification in the image data of machine learning model | |
| JP6216024B1 (en) | Trained model generation method and signal data discrimination device | |
| CN112348737B (en) | Method for generating simulation image, electronic device and storage medium | |
| JP2016517084A (en) | Computer-based method of dynamic category object recognition Computer-based method and system | |
| CN114155546B (en) | Image correction method and device, electronic equipment and storage medium | |
| CN111160110A (en) | Method and device for identifying anchor based on face features and voice print features | |
| CN111127358B (en) | Image processing method, device and storage medium | |
| US20180247431A1 (en) | Process, system and apparatus for machine colour characterisation of digital media | |
| CN109377552B (en) | Image occlusion calculating method, device, calculating equipment and storage medium | |
| CN111260655A (en) | Image generation method and device based on deep neural network model | |
| CN111582654B (en) | Service quality evaluation method and device based on deep cycle neural network | |
| CN111275642B (en) | Low-illumination image enhancement method based on significant foreground content | |
| CN111079744B (en) | Intelligent vehicle license plate identification method and device suitable for complex illumination environment | |
| CN113378864B (en) | Method, device and equipment for determining anchor frame parameters and readable storage medium | |
| CN114359172A (en) | Cigarette carton multi-face detection and identification method and system under stock or display scene | |
| CN111738272A (en) | Target feature extraction method and device and electronic equipment | |
| US20220222836A1 (en) | Method for determining height of plant, electronic device, and storage medium | |
| WO2023035263A1 (en) | Method and device for determining image signal processing parameters, and perception system | |
| CN113066024B (en) | Training method of image blur detection model, image blur detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |