CN115393639B - Intelligent commodity marking method, intelligent commodity marking system, terminal equipment and readable storage medium - Google Patents
Intelligent commodity marking method, intelligent commodity marking system, terminal equipment and readable storage medium Download PDFInfo
- Publication number
- CN115393639B CN115393639B CN202210978554.3A CN202210978554A CN115393639B CN 115393639 B CN115393639 B CN 115393639B CN 202210978554 A CN202210978554 A CN 202210978554A CN 115393639 B CN115393639 B CN 115393639B
- Authority
- CN
- China
- Prior art keywords
- commodity
- model
- unlabeled
- training
- marking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000012549 training Methods 0.000 claims abstract description 76
- 238000013507 mapping Methods 0.000 claims abstract description 21
- 230000008569 process Effects 0.000 claims abstract description 12
- 238000001514 detection method Methods 0.000 claims description 39
- 238000005070 sampling Methods 0.000 claims description 18
- 238000001914 filtration Methods 0.000 claims description 7
- 230000001680 brushing effect Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 230000001502 supplementing effect Effects 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 238000002372 labelling Methods 0.000 description 15
- 239000000047 product Substances 0.000 description 15
- 230000000694 effects Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 3
- 102100040401 DNA topoisomerase 3-alpha Human genes 0.000 description 2
- 101000611068 Homo sapiens DNA topoisomerase 3-alpha Proteins 0.000 description 2
- 101100261006 Salmonella typhi topB gene Proteins 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 101150032437 top-3 gene Proteins 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 235000013361 beverage Nutrition 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 1
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000009469 supplementation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V10/7753—Incorporation of unlabelled data, e.g. multiple instance learning [MIL]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an intelligent commodity marking method, which comprises the steps of randomly selecting M scene pictures from real quick-sales scene pictures, marking commodities to be learned in the scene pictures, and dividing the marked commodities from original scene pictures to obtain n commodity images as training sets; wherein n is less than M, M is the total number of the actual commodity images to be learned in the M scene pictures; dividing unmarked commodities in the M scene pictures to generate unmarked sets; training the improved Mixmatch model by using the training set and the unlabeled set until the model converges, and generating a target classifier; inputting the unlabeled set to a target classifier, and matching the classification result with the mapping table to generate a commodity marking result; the mapping table contains file names and coordinate information of the original pictures where the unlabeled sets are located. According to the method, the accuracy of commodity marking results can be improved by marking a small number of samples, the iteration period is shortened, the marking process is more intelligent, the time consumption is less, and the process is simpler.
Description
Technical Field
The application relates to the technical field of computer software and quick sales, in particular to an intelligent commodity marking method, an intelligent commodity marking system, terminal equipment and a readable storage medium.
Background
With the continuous development of artificial intelligence in the field of quick sales, the large-scale application of target detection in commodity identification has become a trend. The system not only can improve the work efficiency of the salesmen in the shop, but also can help enterprises to quickly know the shop sales details of the terminal shop.
However, in practical applications, a large amount of sample data is required to build an accurate commodity detection model, and a large amount of labor cost and time are required to use a traditional manual marking method before training the model. There are also methods for machine learning assisted marking, such as iterative model strategies, which include training a weak detection model with a lot of data, automatically marking a lot of data by the weak detection model, manually checking and modifying, training a stronger model, and repeating the above steps until the recognition performance meets the requirements. However, the weak detection model in the mode is often poor in classification capability, on one hand, the workload of manual correction is large, and the iteration period is long; on the other hand, this approach is susceptible to missed samples, resulting in undesirable recognition of the final trained model.
Disclosure of Invention
The application aims to provide an intelligent commodity marking method, an intelligent commodity marking system, terminal equipment and a readable storage medium, so as to solve the problems of large workload, long time consumption, low intelligent degree and easiness in influencing the identification accuracy of a commodity detection model in the conventional commodity marking method.
In order to achieve the above purpose, the application provides an intelligent commodity marking method, comprising the following steps:
randomly selecting M scene pictures from the scene pictures of real quick sales, marking commodities to be learned in the scene pictures, and dividing the marked commodities from the original scene pictures to obtain n commodity images serving as training sets; wherein n is less than M, M is the total number of the actual commodity images to be learned in the M scene pictures;
dividing unmarked commodities in M scene pictures to generate unmarked sets;
training an improved Mixmatch model by using the training set and the unlabeled set until the model converges to generate a target classifier;
inputting the unlabeled set to the target classifier, and matching the classification result with a mapping table to generate a commodity marking result; the mapping table contains file name and coordinate information of the original image where the unlabeled set is located.
Further, the matching the classification result with the mapping table to generate a marking result includes:
if the confidence coefficient of the image classification result in the unlabeled set is larger than a first preset value, predicting the current image as a corresponding commodity category, and generating a recognition result;
and matching the identification result with the coordinate information in the mapping table according to the identification sequence, and carrying out information aggregation according to the file name to generate a commodity marking result corresponding to the scene picture.
Further, the training of the modified Mixmatch model includes:
the coarse classifier training stage is used for training to obtain a coarse classifier for classifying unlabeled commodities when the number of the bid products is smaller than a second preset value;
and the coarse classifier optimizing stage is used for manually carrying out random brushing on the data of the coarse classifier, and supplementing the bid data or the missed-identified product data to a training set to optimally train the coarse classifier if the bid is mistakenly identified as the product or the product is missed to be identified in the brushing result.
Further, the dividing the unmarked commodity in the M scene pictures includes:
and dividing unmarked commodities in the M scene pictures by using a commodity detection model, wherein the commodity detection model is obtained by training according to a cascadeRcnn algorithm.
Further, after the goods which are not marked in the M scene pictures are segmented by the goods detection model, the method further comprises:
obtaining a detection result output by a commodity detection model, and calculating the intersection ratio of the detection result and an artificial mark target; the detection result is coordinate information of the input commodity, and the manual marking target is coordinate information of the manual marking commodity;
if the intersection ratio is greater than 0.5, discarding the corresponding scene picture;
and if the intersection ratio is smaller than or equal to 0.5, cutting the commodity image in the corresponding scene picture to serve as an unlabeled set.
Further, before training the modified Mixmatch model with the training set and the unlabeled set, obtaining the modified Mixmatch model further includes:
performing a negative sample filtering strategy, comprising:
inputting the unlabeled set into a target classifier for prediction, and calculating the confidence coefficient of the positive sample and the negative sample;
for positive samples with confidence degrees smaller than the second preset value, the confidence value of the negative samples is improved by using the following formula, and the confidence value of the positive samples is reduced:
wherein ,
f(t)＝min(0.4,0.1+(1-e -3*t/150 )/2)；
where P is the average of multiple branch predictions of the Mixmatch model, P pos Representing the predicted value of the positive sample model, P neg Representing negative sample model predictive value, t represents iteration round, f (t) increases with iteration round.
Further, the obtaining the improved Mixmatch model further includes executing an unlabeled set undersampling strategy, including:
training by using a training set to obtain a Resnet18 classifier model, classifying the unlabeled set by using the Resnet18 classifier model, and outputting a model prediction result of each scene picture;
taking TOP3 to sum according to the model prediction result, and obtaining the score of each scene picture:
score＝sum(top3(P′ pos ))；
in the formula ,P′pos Probability of positive sample class in Resnet18 classifier model output;
sorting the scores, resetting the sampling weight with the score larger than a second preset value to be 1, and calculating the sampling weight with the score smaller than the second preset value by using the following formula:
f(R)＝(R/N cls *4)/(G-R)；
wherein G is the total of the products in the unlabeled setNumber N cls The category number of the commodities to be learned;
and according to the size of the sampling weight, reducing the training process of adding the sample corresponding to the low sampling weight into the Mixmatch model according to a preset condition.
The application also provides an intelligent commodity marking system, which comprises:
the training set generation unit is used for randomly selecting M scene pictures from the scene pictures of real quick sales, marking commodities to be learned in the scene pictures, and dividing the marked commodities from the original scene pictures to obtain n commodity images serving as training sets; wherein n is less than M, M is the total number of the actual commodity images to be learned in the M scene pictures;
the unlabeled set generating unit is used for dividing unlabeled commodities in the M scene pictures to generate unlabeled sets;
the training unit is used for training the improved Mixmatch model by utilizing the training set and the unlabeled set until the model converges to generate a target classifier;
the marking result generating unit is used for inputting the unlabeled set into the target classifier, matching the classification result with the mapping table and generating a commodity marking result; the mapping table contains file name and coordinate information of the original image where the unlabeled set is located.
The application also provides a terminal device, comprising:
one or more processors;
a memory coupled to the processor for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the intelligent marking method for a commodity as described in any one of the preceding claims.
The application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the intelligent marking method for goods as described in any one of the above.
Compared with the prior art, the application has the beneficial effects that:
aiming at the problems of the weak classifier, the application provides the commodity intelligent marking method based on a small amount of sample learning, and the accuracy of intelligent marking can be greatly improved by marking a small amount of samples by fully utilizing the unmarked data, so that the iteration period is accelerated. And the labeling personnel do not need to label all the commodities to be learned in the pictures, so that the labeling process is simpler. Meanwhile, aiming at the problem that the number of negative samples of the mark is small, all other types of commodities which do not need to be learned are difficult to cover, or the problem of data imbalance in the model training process caused by too many standard-free commodities does not need to be learned, the application further provides a negative sample sampling and identifying method, and the accuracy of the model mark bid is effectively improved.
Drawings
In order to more clearly illustrate the technical solutions of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of an intelligent marking method for commodities according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a conventional Mixmatch model according to an embodiment of the present application;
FIG. 3 is a schematic flow chart of a method for intelligent marking of goods according to another embodiment of the present application;
fig. 4 is a schematic structural diagram of an intelligent marking system for commodities according to an embodiment of the present application.
Fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be understood that the step numbers used herein are for convenience of description only and are not limiting as to the order in which the steps are performed.
It is to be understood that the terminology used in the description of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
The terms "comprises" and "comprising" indicate the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The term "and/or" refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Referring to fig. 1, an embodiment of the present application provides an intelligent marking method for a commodity. As shown in fig. 1, the intelligent commodity marking method includes steps S10 to S40. The method comprises the following steps:
s10, randomly selecting M scene pictures from the scene pictures of real quick sales, marking commodities to be learned in the scene pictures, and dividing the marked commodities from the original scene pictures to obtain n commodity images serving as training sets; wherein n is less than M, and M is the total number of the actual commodity images to be learned in the M scene pictures.
In the step, firstly, a group of unlabeled scene picture sets containing a large number of commodity images are collected, wherein the scene refers to a real scene in the quick-sales field, and can be, for example, a quick-sales scene such as a shelf, an end frame, a refrigerator and the like; wherein, a commodity corresponds to a commodity image, and the scene picture contains a plurality of commodity images, namely a plurality of commodities. Typically, to ensure training, as many scene pictures as possible, for example thousands of pictures, the specific number may be chosen according to the actual application environment, without any limitation.
After a large number of scene pictures are obtained, M scene pictures are randomly selected from the scene pictures, and then the commodities to be learned in the scene pictures are marked. After marking, the images are segmented from the original scene images, and then n commodity images are obtained. In this step, let M be the total number of the commodity images to be learned in the M scene pictures, and because only a small number of samples need to be marked in the method, the number of n commodity images generated in this method is far smaller than M.
In a preferred embodiment, since there are only a small number of subsequent annotation images, the sample integrity of the acquired merchandise image is relatively good, e.g., a piece of merchandise has a plurality of different sides, such as a wrapped paper extraction, typically a cuboid, with 6 sides. For such a commodity, images of 6 sides of the commodity need to be acquired when data is acquired.
For example, the frequency of occurrence of any one to-be-learned commodity in the M selected scene pictures in the embodiment should be greater than a certain threshold. Typically, a scene picture contains a large number of commodities, and a commodity may appear multiple times in the scene picture. That is, the number of commodity types in the scene picture may be smaller than the number of commodities.
Assume that the minimum data requirement for each class of commodity that trains a classifier is a threshold T. If 300 pieces of the images are displayed, the frequency of the occurrence of the commodity in the current image set is greater than T, and the frequency is calculated as the total occurrence times of the commodity in all scene images. It should be noted that the threshold is set because the classifier needs to have enough data to train, and the classifier is easy to be over-fitted when the commodity data is insufficient, resulting in insufficient generalized recognition performance.
In one embodiment, marking the goods to be learned in the M scene pictures comprises marking the goods to be learned by using LabelImage.
Specifically, a part of unlabeled scene pictures are selected manually and randomly, and a small amount of commodity images to be learned in the unlabeled scene pictures are marked. If the pictures have the bid products very similar to the commodities to be learned, the pictures are marked as other classes. Here too, the manual labeling is performed by randomly selecting a part of similar commodities. The bid product is a commodity very similar to the bid product, namely, the overall appearance among commodities is very similar, only partial areas are different, the area of the difference area is small, and human eyes cannot distinguish the difference area easily.
It should be noted that, when marking, the number of marks of each type of commodity to be learned should be larger than a certain threshold value, and the number is as balanced as possible. In this embodiment, labeling is mainly performed by a manual labeling tool LabelImage, and the specific labeling operation is partial labeling, that is, partial labeling is performed on the to-be-learned commodity in the scene image roughly, that is, random labeling or non-labeling is adopted, and the number of the partial labeling is far less than the occurrence frequency of the commodity, and at this time, the partial labeling can be considered as a small number of labeling.
Further, after marking, the marked commodity image is scratched to obtain a training set. The matting refers to dividing the commodity picture (sub-picture) from the corresponding position of the original scene picture according to the manually marked result (category and position information). As a preferred embodiment, the training set may also be supplemented by uploading images of the merchandise to be learned and similar bids off-line.
S20, dividing unmarked commodities in the M scene pictures to generate unmarked sets.
In the step, the unmarked commodity in the M scene pictures is segmented by utilizing a commodity detection model, and the commodity detection model is obtained by training according to a cascadeRcnn algorithm.
In a specific embodiment, in order to train a general commodity detection model, a large number of real pictures of quick sales scenes such as shelves, end frames, refrigerators and the like need to be acquired first. And training the real pictures to obtain a general commodity detector which is used for detecting and dividing all commodity pictures. Due to the complexity of real scenes, such as the influences of commodity packaging differences, oblique photographing, illumination and the like, the training set is required to cover various industries of beverages, foods, daily chemicals and the like, and pictures under various imaging conditions are included. In order to ensure the segmentation effect of the commodity detection model, the embodiment preferably collects tens of thousands of picture materials, and trains a commodity detection model which is based on a Cascade Rcnn algorithm and can perform full commodity detection. The detection model is a general detection model and is not specific to specific commodities.
It should be noted that, the Cascade Rcnn algorithm mainly solves the problem that in target detection, a detection frame is not particularly accurate and noise interference is easy to occur. The method is a two-stage detection network, and the aim of continuously optimizing the prediction result is fulfilled by cascading a plurality of detection networks. The cascadeRCNN has the advantages that in the dense shelf images, the detection frame is accurate, and all commodities can be detected well. The accuracy of the detection frame greatly influences the classification effect of the later semi-supervised learning classification.
In an exemplary embodiment, after dividing the unlabeled merchandise in the M scene pictures, to generate the unlabeled set, the method further includes performing the following steps:
1) Obtaining a detection result output by a commodity detection model, and calculating the intersection ratio IOU of the detection result and an artificial mark target; the detection result is coordinate information of the input commodity, and is marked as B1, and the manual marking target is coordinate information of the manual marking commodity, and is marked as B2; therefore, the IOU calculation formula is:
2) If the intersection ratio is greater than 0.5, discarding the corresponding scene picture;
3) And if the intersection ratio is smaller than or equal to 0.5, cutting the commodity image in the corresponding scene picture to serve as an unlabeled set.
S30, training the improved Mixmatch model by using the training set and the unlabeled set until the model converges, and generating a target classifier.
In the step, a training set and an unlabeled set are input into an optimized Mixmatch model for training until a loss function of the model is not reduced in a plurality of rounds, namely the model converges, model parameters of the Mixmatch classifier are stored, and a target classifier is generated.
Referring to fig. 2, a structure of a conventional Mixmatch model is provided. It should be noted that, the traditional Mixmatch model adopts fakelabel strategy to process unlabeled samples, averages the results of two random data enhancement branches, reduces fakelabel noise, and sharpens the output label, so that the unlabeled sample identification result tends to be a specific result, mixup data enhancement can also better process data with certain noise, and the problem of model overfitting is relieved, which is also important in semi-supervised learning.
The efficiency of the whole marking system is directly affected by the performance of the Mixmatch semi-supervised model. If the classification result is poor, multiple rounds of data supplementation are needed, and iterative training is performed. And the mixmatch model faces two key problems in an automatic commodity marking system:
1) The negative sample class (other commodities do not need to be identified) has low correlation among the negative samples and has various forms, but the marked negative sample commodity class is only a very small part of the negative sample commodity class.
2) The problems of multiple negative samples and unbalanced categories lead to the fact that pseudo tags generated by the semi-supervised model contain a large amount of noise.
Thus, the present embodiment makes two improvements over the conventional Mixmatch model to obtain an improved Mixmatch model. The method comprises the following steps:
the first is to perform a negative sample filtering strategy. Because thousands of unlabeled negative sample classes are subjected to learning training through a mixmatch model, classification results of the unlabeled negative sample classes can be divided into positive sample classes in an error manner, and errors are continuously transmitted along with network iterative training, so that the final model classification effect is affected. For this, we need to adjust the model output of the possible negative samples during the training of the model, the adjustment strategy is as follows: aiming at the samples which are judged to be positive samples and have low confidence, the confidence value of the negative samples is improved, and the confidence value of the positive samples is reduced.
Specifically, a negative sample filtering strategy is performed, comprising the steps of:
inputting the unlabeled set into a target classifier for prediction, and calculating the confidence coefficient of the positive sample and the negative sample;
for positive samples with confidence degrees smaller than the second preset value, the confidence value of the negative samples is improved by using the following formula, and the confidence value of the positive samples is reduced:
wherein ,
f(t)＝min(0.4,0.1+(1-e -3*t/150 )/2)；
where P is the average of multiple branch predictions of the Mixmatch model, P pos Representing the predicted value of the positive sample model, P neg Representing negative sample model predictive value, t represents iteration round, f (t) increases with iteration round.
And secondly, executing an undersampling strategy when negative sample data are excessive, wherein the undersampling strategy comprises the following steps of:
training by using a training set to obtain a Resnet18 classifier model, classifying the unlabeled set by using the Resnet18 classifier model, and outputting a model prediction result of each scene picture;
taking TOP3 to sum according to the model prediction result, and obtaining the score of each scene picture:
score＝sum(top3(P′ pos ))；
in the formula ,P′pos Probability of positive sample class in Resnet18 classifier model output;
sorting the scores, and pre-estimating the display quantity value R of a product on all shelves in advance to be used as a second preset value; and (3) arranging the pictures with scores in front of R, namely, setting the sampling weight with scores larger than a second preset value to be 1, and calculating the sampling weight with scores smaller than the second preset value by using the following formula:
f(R)＝(R/N cls *4)/(G-R)；
wherein G is the total number of the commodities in the unlabeled set, N cls Is the category number of the commodity to be learned.
And according to the size of the sampling weight, reducing the training process of adding the sample corresponding to the low sampling weight into the Mixmatch model according to a preset condition.
S40, inputting the unlabeled set into the target classifier, and matching the classification result with a mapping table to generate a commodity marking result; the mapping table contains file name and coordinate information of the original image where the unlabeled set is located.
In the step, firstly, an unlabeled set is input to a target classifier for classification, and then a classification result is matched with a mapping table to generate a commodity marking result, which comprises the following steps:
if the confidence coefficient of the image classification result in the unlabeled set is larger than a first preset value, predicting the current image as a corresponding commodity category, and generating a recognition result;
and matching the identification result with the coordinate information in the mapping table according to the identification sequence, and carrying out information aggregation according to the file name to generate a commodity marking result corresponding to the scene picture.
In this embodiment, if the confidence coefficient of the classification result of a certain scene picture is greater than a first preset value, predicting the current scene picture as the commodity category to be learned; namely, the following conditions are satisfied:
cls＝argmax(pred)s.t.max(softmax(pred))>T s ；
wherein cls is the prediction category, softmax (pred) is the confidence, pred is the output of the target classifier, T s Is a first preset value.
And matching the current scene picture with the mapping table to obtain file name and coordinate information of the original picture of the commodity, and generating a commodity marking result.
It will be appreciated that the MAP table (MAP) herein is used to hold location coordinate information of commodity subgraph scene pictures. And after the subgraph is scratched out, the subgraph enters a classifier, and the output category of the classifier and the position information in the MAP form a marking output result together. And generating a corresponding original image mark file through the classification result of the classifier and the information in the corresponding MAP, wherein the position information of the frame is derived from a general commodity detection model, and the class is the output class of the target classifier.
However, in practical application, there is a case that when a trained classifier is used to classify an unlabeled set, a situation that a certain bid product is recognized as a commodity to be learned by a lot of errors or other angle images of a commodity image to be learned of the commodity to be learned are not recognized correctly occurs. It can be stated that the classification effect of the target classifier trained in step S30 is still to be improved.
To address this problem, in one particular embodiment, training the modified Mixmatch model consists essentially of:
the coarse classifier training stage is used for training to obtain a coarse classifier for classifying unlabeled commodities when the number of the bid products is smaller than a second preset value;
and the coarse classifier optimizing stage is used for manually carrying out random brushing on the data of the coarse classifier, and supplementing the bid data or the missed-identified product data to a training set to optimally train the coarse classifier if the bid is mistakenly identified as the product or the product is missed to be identified in the brushing result.
In this embodiment, after the unlabeled set is input to the target classifier, it is generally required to manually and roughly check the classification result of the unlabeled set input to the target classifier, and then supplement a small amount of error data to the training set to optimize the Mixmatch model, as shown in fig. 3. The method specifically comprises the following steps:
when the bid product is mistakenly recognized as the commodity to be learned or other angle images of the commodity image to be learned are not correctly recognized, respectively supplementing the mistaken recognition image and the other angle images of the commodity image to be learned into a training set;
iteratively training the target classifier by using the supplemented training set and the unlabeled set until the model converges to obtain an optimized target classifier;
and inputting the unlabeled set into an optimized target classifier to classify.
In this embodiment, if a bid item is mistakenly identified as a to-be-learned commodity, a small number of mistakenly identified commodity images are added to other classes, for example, a small number of pictures s < m (the total number of occurrences of a bid item in a scene image set) can be selected, and s is typically 2 to 3.
Similarly, if other angle images of the commodity image to be learned are not correctly identified, for example, a certain side image is not correctly identified or is not correctly identified, a small amount of side images are correspondingly supplemented to the training set of the corresponding category. Preferably, the replenishment number is less than 10.
Further, the method for rough verification in the implementation specifically comprises the following steps: and classifying the results in the unlabeled set into different sets according to the prediction category, sorting each set according to the confidence score in sequence, and selecting the part with the confidence level lower than 0.95 and higher than 0.7 for manual verification. It will be appreciated that confidence levels below 0.95 and above 0.7 are merely preferred, and that the size of the numerical ranges may be adjusted as desired and are not limited in any way herein.
In summary, the embodiment of the application provides the commodity intelligent marking method based on a small amount of sample learning aiming at the problems of the weak classifier, and the accuracy of intelligent marking can be greatly improved by marking a small amount of samples by fully utilizing the unmarked data and further accelerating the iteration period. And the labeling personnel do not need to label all the commodities to be learned in the pictures, so that the labeling process is simpler. Meanwhile, aiming at the problem that the number of negative samples of the mark is small, all other types of commodities which do not need to be learned are difficult to cover, or the problem of data imbalance in the model training process caused by too many standard-free commodities does not need to be learned, the application further provides a negative sample sampling and identifying method, and the accuracy of the model mark bid is effectively improved.
Referring to fig. 4, an embodiment of the present application further provides an intelligent marking system for a commodity, including:
the training set generating unit 01 is used for randomly selecting M scene pictures from the scene pictures of real quick sales, marking commodities to be learned in the scene pictures, and dividing the marked commodities from the original scene pictures to obtain n commodity images serving as training sets; wherein n is less than M, M is the total number of the actual commodity images to be learned in the M scene pictures;
the unlabeled set generating unit 02 is used for dividing unlabeled commodities in the M scene pictures to generate unlabeled sets;
the training unit 03 is configured to train the improved Mixmatch model by using the training set and the unlabeled set until the model converges, so as to generate a target classifier;
the marking result generating unit 04 is used for inputting the unlabeled set to the target classifier, matching the classification result with the mapping table and generating a commodity marking result; the mapping table contains file name and coordinate information of the original image where the unlabeled set is located.
It can be understood that the intelligent commodity marking system provided in this embodiment is configured to execute the intelligent commodity marking method according to any one of the foregoing embodiments, and achieve the same effects as the foregoing, which will not be further described herein.
Referring to fig. 5, an embodiment of the present application provides a terminal device, including:
one or more processors;
a memory coupled to the processor for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the intelligent marking method for goods as described above.
The processor is used for controlling the whole operation of the terminal equipment so as to complete all or part of the steps of the intelligent commodity marking method. The memory is used to store various types of data to support operation at the terminal device, which may include, for example, instructions for any application or method operating on the terminal device, as well as application-related data. The Memory may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as static random access Memory (Static Random Access Memory, SRAM for short), electrically erasable programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), erasable programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM for short), programmable Read-Only Memory (Programmable Read-Only Memory, PROM for short), read-Only Memory (ROM for short), magnetic Memory, flash Memory, magnetic disk or optical disk.
In an exemplary embodiment, the terminal device may be implemented by one or more application specific integrated circuits (Application Specific 1ntegrated Circuit, abbreviated AS 1C), digital signal processor (Digital Signal Processor, abbreviated DSP), digital signal processing device (Digital Signal Processing Device, abbreviated DSPD), programmable logic device (Programmable Logic Device, abbreviated PLD), field programmable gate array (Field Programmable Gate Array, abbreviated FPGA), controller, microcontroller, microprocessor or other electronic component, for performing the method for intelligent marking of goods according to any of the foregoing embodiments, and achieving technical effects consistent with the foregoing method.
In another exemplary embodiment, there is also provided a computer readable storage medium comprising a computer program which, when executed by a processor, implements the steps of the intelligent marking method for an article as described in any of the embodiments above. For example, the computer readable storage medium may be a memory including a computer program, where the computer program is executable by a processor of a terminal device to perform the intelligent commodity marking method according to any one of the embodiments, and achieve technical effects consistent with the method.
Finally, it should be noted that the above embodiments are merely illustrative of the technical solution of the present application, and not limiting thereof; while the application has been described in detail with reference to the foregoing embodiments, it will be appreciated by those skilled in the art that variations may be made in the techniques described in the foregoing embodiments, or equivalents may be substituted for elements thereof; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims (8)
1. An intelligent commodity marking method is characterized by comprising the following steps:
random selection from real quick-sales scene picturesA scene picture, for whichMarking the goods to be learned, and dividing the marked goods from the original scene image to obtain +.>The commodity image is used as a training set; wherein (1)>，/>Is->The total number of the actual commodity images to be learned in the scene pictures;
will beDividing unlabeled commodities in a scene picture to generate unlabeled sets;
training an improved Mixmatch model by using the training set and the unlabeled set until the model converges to generate a target classifier;
inputting the unlabeled set to the target classifier, and matching the classification result with a mapping table to generate a commodity marking result; the mapping table comprises file name and coordinate information of the original image where the unlabeled set is located;
the improved Mixmatch model is an improvement on the traditional Mixmatch model in two aspects, namely, a negative sample filtering strategy is executed, and an unlabeled set undersampling strategy is executed;
the performing a negative sample filtering policy includes:
inputting the unlabeled set into a target classifier for prediction, and calculating the confidence coefficient of the positive sample and the negative sample;
for positive samples with confidence degrees smaller than the second preset value, the confidence value of the negative samples is improved by using the following formula, and the confidence value of the positive samples is reduced:
；
wherein ,
；
where P is the average of the branch predictions of the Mixmatch model,representing positive sample model predictive value,/->Representing negative sample model predictive value, t representing iteration round,/->With the iteration rounds increasing continuously;
the executing of the unlabeled set undersampling strategy includes:
training by using a training set to obtain a Resnet18 classifier model, classifying the unlabeled set by using the Resnet18 classifier model, and outputting a model prediction result of each scene picture;
taking according to the model prediction resultAnd 3, summing to obtain the score of each scene picture:；
in the formula ,probability of positive sample class in Resnet18 classifier model output;
sorting the scores, resetting the sampling weight with the score larger than a second preset value to be 1, and calculating the sampling weight with the score smaller than the second preset value by using the following formula:
；
wherein G is the total number of the commodities in the unlabeled set,the category number of the commodities to be learned;
and according to the size of the sampling weight, reducing the training process of adding the sample corresponding to the low sampling weight into the Mixmatch model according to a preset condition.
2. The intelligent commodity marking method according to claim 1, wherein the matching the classification result with the mapping table to generate a marking result comprises:
if the confidence coefficient of the image classification result in the unlabeled set is larger than a first preset value, predicting the current image as a corresponding commodity category, and generating a recognition result;
and matching the identification result with the coordinate information in the mapping table according to the identification sequence, and carrying out information aggregation according to the file name to generate a commodity marking result corresponding to the scene picture.
3. The intelligent commodity marking method according to claim 1, wherein the training of the modified Mixmatch model comprises:
the coarse classifier training stage is used for training to obtain a coarse classifier for classifying unlabeled commodities when the number of the bid products is smaller than a second preset value;
and the coarse classifier optimizing stage is used for manually carrying out random brushing on the data of the coarse classifier, and supplementing the bid data or the missed-identified product data to a training set to optimally train the coarse classifier if the bid is mistakenly identified as the product or the product is missed to be identified in the brushing result.
4. The intelligent marking method according to claim 1, wherein the commodity is to be markedThe method for dividing the unlabeled commodity in the scene picture comprises the following steps:
using commodity detection modelAnd dividing the unlabeled commodity in the scene picture, and training the commodity detection model according to a cascadeRcnn algorithm.
5. The intelligent commodity marking method according to claim 4, wherein said commodity detection model is used for the commodity markingAfter the unmarked commodity in the scene picture is segmented, the method further comprises the following steps:
obtaining a detection result output by a commodity detection model, and calculating the intersection ratio of the detection result and an artificial mark target; the detection result is coordinate information of the input commodity, and the manual marking target is coordinate information of the manual marking commodity;
if the intersection ratio is greater than 0.5, discarding the corresponding scene picture;
and if the intersection ratio is smaller than or equal to 0.5, cutting the commodity image in the corresponding scene picture to serve as an unlabeled set.
6. An intelligent marking system for goods, comprising:
the training set generating unit is used for randomly selecting from the scene pictures of real quick salesMarking the commodity to be learned in the scene picture, and dividing the marked commodity from the original scene picture to obtain +.>The commodity image is used as a training set; wherein (1)>Is->The total number of the actual commodity images to be learned in the scene pictures;
an unlabeled set generation unit for generatingDividing unlabeled commodities in a scene picture to generate unlabeled sets;
the training unit is used for training the improved Mixmatch model by utilizing the training set and the unlabeled set until the model converges to generate a target classifier; the improved Mixmatch model is an improvement on the traditional Mixmatch model in two aspects, namely, a negative sample filtering strategy is executed, and an unlabeled set undersampling strategy is executed;
the performing a negative sample filtering policy includes:
inputting the unlabeled set into a target classifier for prediction, and calculating the confidence coefficient of the positive sample and the negative sample;
for positive samples with confidence degrees smaller than the second preset value, the confidence value of the negative samples is improved by using the following formula, and the confidence value of the positive samples is reduced:
；
wherein ,
；
where P is the average of the branch predictions of the Mixmatch model,representing positive sample model predictive value,/->Representing negative sample model predictive value, t representing iteration round,/->With the iteration rounds increasing continuously;
the executing of the unlabeled set undersampling strategy includes:
training by using a training set to obtain a Resnet18 classifier model, classifying the unlabeled set by using the Resnet18 classifier model, and outputting a model prediction result of each scene picture;
taking according to the model prediction resultAnd 3, summing to obtain the score of each scene picture:；
in the formula ,probability of positive sample class in Resnet18 classifier model output;
sorting the scores, resetting the sampling weight with the score larger than a second preset value to be 1, and calculating the sampling weight with the score smaller than the second preset value by using the following formula:
；
wherein G is the total number of the commodities in the unlabeled set,the category number of the commodities to be learned;
according to the size of the sampling weight, reducing the training process of adding samples corresponding to the low sampling weight into the Mixmatch model according to a preset condition;
the marking result generating unit is used for inputting the unlabeled set into the target classifier, matching the classification result with the mapping table and generating a commodity marking result; the mapping table contains file name and coordinate information of the original image where the unlabeled set is located.
7. A terminal device, comprising:
one or more processors;
a memory coupled to the processor for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the intelligent marking method of an item as claimed in any one of claims 1 to 5.
8. A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the intelligent marking method for goods as claimed in any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210978554.3A CN115393639B (en) | 2022-08-16 | 2022-08-16 | Intelligent commodity marking method, intelligent commodity marking system, terminal equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210978554.3A CN115393639B (en) | 2022-08-16 | 2022-08-16 | Intelligent commodity marking method, intelligent commodity marking system, terminal equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115393639A CN115393639A (en) | 2022-11-25 |
| CN115393639B true CN115393639B (en) | 2023-08-11 |
Family
ID=84121240
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210978554.3A Active CN115393639B (en) | 2022-08-16 | 2022-08-16 | Intelligent commodity marking method, intelligent commodity marking system, terminal equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115393639B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116128954B (en) * | 2022-12-30 | 2023-12-05 | 上海强仝智能科技有限公司 | Commodity layout identification method, device and storage medium based on generation network |
| CN116311219B (en) * | 2023-01-29 | 2024-02-09 | 广州市玄武无线科技股份有限公司 | Ground pile article occupied area calculation method and device based on perspective transformation |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2018026119A (en) * | 2016-07-29 | 2018-02-15 | 株式会社野村総合研究所 | Classification system, control method of classification system, and program |
| CN107871144A (en) * | 2017-11-24 | 2018-04-03 | 税友软件集团股份有限公司 | Invoice trade name sorting technique, system, equipment and computer-readable recording medium |
| WO2019062811A1 (en) * | 2017-09-27 | 2019-04-04 | 缤果可为（北京）科技有限公司 | Automatic image acquisition and labeling device and method |
| JP2019212157A (en) * | 2018-06-07 | 2019-12-12 | 大日本印刷株式会社 | Commodity specification device, program, and learning method |
| CN111723209A (en) * | 2020-06-28 | 2020-09-29 | 上海携旅信息技术有限公司 | Semi-supervised text classification model training method, text classification method, system, device and medium |
| CN112257767A (en) * | 2020-10-16 | 2021-01-22 | 浙江大学 | Product key part state classification method aiming at class imbalance data |
| WO2021046951A1 (en) * | 2019-09-09 | 2021-03-18 | 安徽继远软件有限公司 | Image identification method, system, and storage medium |
| CN113850249A (en) * | 2021-12-01 | 2021-12-28 | 深圳市迪博企业风险管理技术有限公司 | Method for formatting and extracting chart information |
| CN113869211A (en) * | 2021-09-28 | 2021-12-31 | 杭州福柜科技有限公司 | Automatic image annotation and automatic annotation quality evaluation method and system |
| CN114186056A (en) * | 2021-12-14 | 2022-03-15 | 广州华多网络科技有限公司 | Commodity label labeling method and device, equipment, medium and product thereof |
| CN114529351A (en) * | 2022-03-10 | 2022-05-24 | 上海微盟企业发展有限公司 | Commodity category prediction method, device, equipment and storage medium |
-
2022
- 2022-08-16 CN CN202210978554.3A patent/CN115393639B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2018026119A (en) * | 2016-07-29 | 2018-02-15 | 株式会社野村総合研究所 | Classification system, control method of classification system, and program |
| WO2019062811A1 (en) * | 2017-09-27 | 2019-04-04 | 缤果可为（北京）科技有限公司 | Automatic image acquisition and labeling device and method |
| CN107871144A (en) * | 2017-11-24 | 2018-04-03 | 税友软件集团股份有限公司 | Invoice trade name sorting technique, system, equipment and computer-readable recording medium |
| JP2019212157A (en) * | 2018-06-07 | 2019-12-12 | 大日本印刷株式会社 | Commodity specification device, program, and learning method |
| WO2021046951A1 (en) * | 2019-09-09 | 2021-03-18 | 安徽继远软件有限公司 | Image identification method, system, and storage medium |
| CN111723209A (en) * | 2020-06-28 | 2020-09-29 | 上海携旅信息技术有限公司 | Semi-supervised text classification model training method, text classification method, system, device and medium |
| CN112257767A (en) * | 2020-10-16 | 2021-01-22 | 浙江大学 | Product key part state classification method aiming at class imbalance data |
| CN113869211A (en) * | 2021-09-28 | 2021-12-31 | 杭州福柜科技有限公司 | Automatic image annotation and automatic annotation quality evaluation method and system |
| CN113850249A (en) * | 2021-12-01 | 2021-12-28 | 深圳市迪博企业风险管理技术有限公司 | Method for formatting and extracting chart information |
| CN114186056A (en) * | 2021-12-14 | 2022-03-15 | 广州华多网络科技有限公司 | Commodity label labeling method and device, equipment, medium and product thereof |
| CN114529351A (en) * | 2022-03-10 | 2022-05-24 | 上海微盟企业发展有限公司 | Commodity category prediction method, device, equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 许重建等.基于深度学习的HS Code产品归类方法研究.《现代计算机(专业版)》.2019,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115393639A (en) | 2022-11-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115393639B (en) | Intelligent commodity marking method, intelligent commodity marking system, terminal equipment and readable storage medium | |
| CN111340126B (en) | Article identification method, apparatus, computer device, and storage medium | |
| TWI726364B (en) | Computer-executed vehicle damage assessment method and device | |
| Stalder et al. | Beyond semi-supervised tracking: Tracking should be as simple as detection, but not simpler than recognition | |
| Esmaeili et al. | Fast-at: Fast automatic thumbnail generation using deep neural networks | |
| US20210374472A1 (en) | Systems and methods for creating training data | |
| CN111652317B (en) | Super-parameter image segmentation method based on Bayes deep learning | |
| US7643674B2 (en) | Classification methods, classifier determination methods, classifiers, classifier determination devices, and articles of manufacture | |
| CN111274981B (en) | Target detection network construction method and device and target detection method | |
| CN111178446B (en) | Optimization method and device of target classification model based on neural network | |
| US11599983B2 (en) | System and method for automated electronic catalogue management and electronic image quality assessment | |
| JP2018512567A (en) | Barcode tag detection in side view sample tube images for laboratory automation | |
| CN109977824B (en) | Article taking and placing identification method, device and equipment | |
| CN111339887A (en) | Commodity identification method and intelligent container system | |
| CN113902944A (en) | Model training and scene recognition method, device, equipment and medium | |
| CN115344805A (en) | Material auditing method, computing equipment and storage medium | |
| CN107403179A (en) | A kind of register method and device of article packaged information | |
| CN112232288A (en) | Satellite map target identification method and system based on deep learning | |
| CN111242094A (en) | Commodity identification method, intelligent container and intelligent container system | |
| CN110580299B (en) | Method, system, equipment and storage medium for generating matching diagram of recommended language of object | |
| CN113947771B (en) | Image recognition method, apparatus, device, storage medium, and program product | |
| CN112508659B (en) | Commodity settlement processing method and device, computing equipment and computer storage medium | |
| CN113408528A (en) | Commodity image quality identification method and device, computing equipment and storage medium | |
| Nair et al. | Automation of Cricket Scoreboard by Recognizing Umpire Gestures | |
| Pieropan et al. | Robust and adaptive keypoint-based object tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |