CN111368942B - Commodity classification identification method and device, electronic equipment and storage medium - Google Patents
Commodity classification identification method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111368942B CN111368942B CN202010458232.7A CN202010458232A CN111368942B CN 111368942 B CN111368942 B CN 111368942B CN 202010458232 A CN202010458232 A CN 202010458232A CN 111368942 B CN111368942 B CN 111368942B
- Authority
- CN
- China
- Prior art keywords
- image
- area
- commodity
- attention
- classification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 239000013598 vector Substances 0.000 claims abstract description 43
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 26
- 238000011176 pooling Methods 0.000 claims abstract description 18
- 238000013528 artificial neural network Methods 0.000 claims abstract description 8
- 238000000605 extraction Methods 0.000 claims description 33
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012856 packing Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The application provides a commodity classification and identification method and device, electronic equipment and a storage medium. The commodity classification and identification method comprises the following steps: extracting first characteristic information of each area image of a commodity distribution image by adopting a first convolution neural network, wherein the commodity distribution image comprises a plurality of mutually-disjoint area images, and each area image corresponds to a commodity; generating a corresponding attention area map according to the area image and the corresponding first characteristic information; extracting second characteristic information of the attention area map by adopting a second convolutional neural network; pooling the first characteristic information and the second characteristic information to obtain a bilinear vector; and obtaining the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vector, and obtaining a classification result based on the probability distribution condition. According to the method and the device, the fine characteristic difference is identified by adopting the bilinear vector, so that the accuracy of commodity classification identification can be improved.
Description
Technical Field
The application relates to the technical field of container commodity identification, in particular to a commodity classification identification method and device, electronic equipment and a storage medium.
Background
In the open type intelligent retail container, the fisheye camera is adopted to collect images of commodities placed in the container, so that the commodities are classified and identified. For the container operator, a higher classification recognition accuracy is essential.
The fisheye camera has shelter from when shooing the formation of image to commodity, and the image of its shooting shows the top more and is close to the regional local image in top, lacks the more prominent characteristic information of other parts of commodity. Therefore, this situation affects the accuracy of the classification recognition algorithm, resulting in a decrease in the accuracy of the classification recognition.
In view of the above problems, no effective technical solution exists at present.
Disclosure of Invention
An object of the embodiments of the present application is to provide a method, an apparatus, an electronic device, and a storage medium for classifying and identifying a commodity, which can improve the accuracy of the commodity classification and identification.
In a first aspect, an embodiment of the present application provides a method for classifying and identifying a commodity, including:
extracting first characteristic information of each area image of a commodity distribution image by adopting a first convolution neural network, wherein the commodity distribution image comprises a plurality of mutually-disjoint area images, and each area image corresponds to a commodity;
generating a corresponding attention area map according to the area image and the corresponding first characteristic information;
extracting second characteristic information of the attention area map by adopting a second convolutional neural network;
pooling the first characteristic information and the corresponding second characteristic information to obtain a bilinear vector;
and obtaining the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vector, and obtaining a classification result based on the probability distribution condition.
According to the embodiment of the application, the attention area image with more detailed information such as details and textures is subjected to feature extraction, and the feature information of the area image is combined, so that the fine feature difference of the commodity is identified by adopting the bilinear vector, and the accuracy of commodity classification identification can be improved.
Optionally, in the method for classifying and identifying a product according to the embodiment of the present application, the generating a corresponding attention area map according to the area image and the corresponding first feature information includes:
inputting the first characteristic information into a preset attention area extraction model to acquire position information of an attention area;
cutting the corresponding region image according to the position information of the attention region to obtain an initial attention region image;
and performing up-sampling processing on the initial attention area map to acquire an attention area map with the same resolution as that of a corresponding area image.
Optionally, in the method for classifying and identifying a commodity according to the embodiment of the present application, the region image is rectangular, and the attention region is square;
the cutting the corresponding region image according to the position information of the attention region to obtain an initial attention region map includes:
generating a mask M, wherein the size and the shape of the mask M are the same as those of the area image, and coordinate points of the mask M correspond to coordinate points of the area image one to one respectively;
acquiring a constraint calculation formula of the coordinate point of the mask M and the position information of the attention area;
and cutting out the part of the area image, which is positioned in the attention area, according to the constraint calculation formula to obtain an initial attention area map.
Optionally, in the method for classifying and identifying a commodity according to the embodiment of the present application, the preset attention area extraction model includes at least two convolution layers connected in sequence.
Optionally, in the method for classifying and identifying a commodity according to the embodiment of the present application, the obtaining a probability distribution situation of a classification of the commodity corresponding to the region image according to the bilinear vector, and obtaining a classification result based on the probability distribution situation includes:
sequentially carrying out square root taking operation and L2 normalization operation on the bilinear vectors to obtain target vectors;
inputting the target vector into a softmax function to obtain the probability distribution condition of the classification of the commodity corresponding to the area image;
and obtaining a classification result corresponding to the commodity corresponding to the area image according to the probability distribution condition.
Optionally, in the method for classifying and identifying a commodity according to the embodiment of the present application, before the extracting, by using a first convolutional neural network, first feature information of each area image of a commodity distribution image, the method further includes:
acquiring a commodity distribution image, and generating a plurality of calibration frames in the commodity distribution image, wherein each calibration frame comprises a commodity, and the calibration frames are not intersected with each other;
and extracting the image of the area surrounded by each calibration frame to obtain a corresponding area image.
In a second aspect, an embodiment of the present application further provides a device for classifying and identifying a commodity, including:
the system comprises a first extraction module, a second extraction module and a third extraction module, wherein the first extraction module is used for extracting first characteristic information of each area image of a commodity distribution image by adopting a first convolution neural network, the commodity distribution image comprises a plurality of mutually-disjoint area images, and each area image corresponds to a commodity;
the first generation module is used for generating a corresponding attention area map according to the area image and corresponding first feature information;
the second extraction module is used for extracting second characteristic information of the attention area map by adopting a second convolutional neural network;
the pooling module is used for pooling the first characteristic information and the second characteristic information corresponding to the first characteristic information to obtain a bilinear vector;
and the identification module is used for acquiring the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vector and acquiring a classification result based on the probability distribution condition.
Optionally, in the article classification and identification device according to an embodiment of the present application, the first generating module includes:
a first obtaining unit, configured to input the first feature information into a preset attention area extraction model to obtain location information of an attention area;
the second acquisition unit is used for cutting the corresponding region image according to the position information of the attention region so as to acquire an initial attention region map;
an amplifying unit for performing up-sampling processing on the initial attention area map to acquire an attention area map having the same resolution as that of the corresponding area image
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method as provided in the first aspect are executed.
In a fourth aspect, embodiments of the present application provide a storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the steps in the method as provided in the first aspect.
As can be seen from the above, in the embodiment of the application, the first feature information of each area image of the commodity distribution image is extracted by using the first convolutional neural network, where the area image includes a plurality of mutually disjoint area images, and each area image corresponds to a commodity; generating a corresponding attention area map according to the area image and the corresponding first characteristic information; extracting second characteristic information of the attention area map by adopting a second convolutional neural network; pooling the first characteristic information and the second characteristic information to obtain a bilinear vector; acquiring the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vector, and acquiring a classification result based on the probability distribution condition; thereby realizing the classification and identification of the commodities; according to the method and the device, the attention area image with more detailed information such as details and textures is subjected to feature extraction, and the feature information of the area image is combined, so that the fine feature difference of the commodity is identified by adopting the bilinear vector, and the accuracy of commodity classification identification can be improved.
The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a flowchart of a method for identifying a product classification according to an embodiment of the present application.
Fig. 2 is a schematic structural diagram of a product classification and identification device according to an embodiment of the present application.
Fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Referring to fig. 1, fig. 1 is a flowchart illustrating a product classification and identification method according to some embodiments of the present disclosure. The commodity classification and identification method is used for identifying the classification of commodities in the open type intelligent retail container so as to facilitate automatic generation of orders or automatic settlement in an automatic selling process. The commodity classification and identification method can be applied to a remote server in communication connection with the open type intelligent retail container and can also be applied to a main control module arranged in the open type intelligent retail container. The commodity classification and identification method comprises the following steps:
s101, extracting first characteristic information of an area image of a commodity distribution image by using a first convolution neural network, wherein the commodity distribution image comprises a plurality of mutually-disjoint area images, and each area image corresponds to a commodity.
And S102, generating a corresponding attention area map according to the area image and the corresponding first characteristic information.
S103, extracting second characteristic information of the attention area map by adopting a second convolutional neural network.
And S104, performing pooling processing on the first characteristic information and the corresponding second characteristic information to obtain a bilinear vector.
And S105, obtaining the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vectors, and obtaining a classification result based on the probability distribution condition.
In step S101, the distribution image of the product in the open intelligent retail container is captured by the fisheye camera in the open intelligent retail container. The region image is rectangular or square, each region image corresponds to an image of a commodity, and the plurality of region images are the same in size. When arranging commodity, in order to be convenient for shoot and be convenient for access, a plurality of commodities in this packing cupboard are the rectangular array interval and arrange.
Wherein, a preset first convolution neural network a can be adopted to perform feature extraction on each region image X to obtain a corresponding feature fA(X). The first convolutional neural network a is a convolutional neural network commonly used in the prior art for extracting picture feature information.
It is understood that, in some embodiments, before the step S101, the method for identifying the product classification further includes the following steps: s1011, acquiring a commodity distribution image, and generating a plurality of calibration frames in the commodity distribution image, wherein each calibration frame comprises a commodity, and the calibration frames are not intersected with each other; and S1012, extracting the image of the area surrounded by the calibration frames to obtain a corresponding area image. The calibration frames are rectangular, and any two adjacent calibration frames do not intersect. Then, an image in which each of the calibration frames encloses an area is extracted from the product distribution image with the calibration frame as a boundary, thereby obtaining a plurality of area images.
In step S102, the attention area map is an image of an area where the first feature information is extracted from the area image. Wherein, the attention area graph and the attention area are both square. In some embodiments, this step S102 includes the following sub-steps: s1021, inputting the first feature information into a preset attention area extraction model to obtain position information of an attention area; s1022, cutting the corresponding area image according to the position information of the attention area to acquire an initial attention area image; and S1023, performing up-sampling processing on the initial attention area map to acquire an attention area map with the same size and resolution as the corresponding area image.
In step S1021, the predetermined attention area extraction model is composed of at least two convolution layers, which may be two or more convolution layers. The position information of the attention area is { tx, ty, t1}, where (tx, ty) is the center coordinate of the attention area, the t1 is half the side length of the attention area, and the attention area defaults to a square.
In step S1022, the clipped attention area image is rectangular, and in the specific operation, a rectangular area of all the first extracted feature information is extracted from the corresponding area image according to the position information, that is, all the extracted first feature information is extracted from the rectangular area. Therefore, after the clipping, the ratio of the number of the first feature information in the attention area map to the area of the attention area map is increased, which is convenient for extracting and identifying feature information such as details and textures in the attention area map.
Specifically, in some embodiments, this step S1021 includes: generating a mask M, wherein the size and the shape of the mask M are the same as those of the area image, and each coordinate point of the mask M corresponds to each coordinate point of the area image one to one; acquiring a constraint calculation formula of the coordinate point of the mask M and the position information of the attention area; according to said orderThe bundle calculation formula cuts out a portion of the region image located in the attention region to obtain an initial attention region map. Wherein, the constraint calculation formula is M (x, y) = { h (x-tx 1) -h (x-tx 2) } { h (y-ty 1) -h (y-ty 2) }; wherein h (x) = 1/(1 + e)-kx). Where, (tx 1, ty 1) is the coordinate of the top left corner of the attention area obtained from { tx, ty, t1} as the position information, and (tx 2, ty 2) is the coordinate of the bottom right corner of the attention area obtained from { tx, ty, t1} as the position information. By setting the value of the empirical value k, it can be ensured that the value of the mask M (x, y) is 1 when only the coordinate point of the mask M is in the attention area, and the value of the mask M (x, y) is 0 when not in the attention area; the M (x, y) is thus used to crop the initial attention area map from the corresponding area image.
In this step S103, a preset second convolutional neural network B may be used to perform feature extraction on each attention area map Y to obtain a corresponding feature fB(Y). The second convolutional neural network B is a convolutional neural network commonly used in the prior art for extracting picture feature information. The attention area map in the area image is cut out, and the attention area map is set to have the same size and resolution as the area image, so that the detail and texture information of the commodity can be conveniently extracted from the attention area map, and the accuracy of subsequent classification and identification is improved.
In the step S104, the first characteristic information f is first obtainedA(X) and the second characteristic information fBAnd (Y) merging, and then performing pooling operation on the merged feature information by adopting a preset pooling layer to obtain a bilinear feature vector V.
In step S105, the bilinear vector V may be preprocessed and then input into a preset probability distribution prediction function to obtain a probability distribution of the classification of the commodity corresponding to the region image.
Specifically, in some embodiments, this step S105 includes: s1051, sequentially carrying out square root taking operation and L2 normalization operation on the bilinear vectors to obtain target vectors; s1052, inputting the target vector into a softmax function to obtain the probability distribution condition of the classification of the commodity corresponding to the area image; and S1053, obtaining the classification result corresponding to the commodity corresponding to the area image according to the probability distribution condition. Wherein, the probability distribution situation is used for describing the probability that the commodity corresponding to the area image belongs to various categories. For example, for a certain region image, the probability distribution obtained at the end is: p1, P2, P3, P4 and P5, wherein the commodity categories corresponding to P1, P2, P3, P4 and P5 are A1, A2, A3, A4 and A5 respectively. Among these P1, P2, P3, P4, and P5, since P2 is the largest, the classification of the certain region image is a2 type.
As can be seen from the above, in the embodiment of the application, the first feature information of each area image of the commodity distribution image is extracted by using the first convolutional neural network, where the area image includes a plurality of mutually disjoint area images, and each area image corresponds to a commodity; generating a corresponding attention area map according to the area image and the corresponding first characteristic information; extracting second characteristic information of the attention area map by adopting a second convolutional neural network; pooling the first characteristic information and the second characteristic information to obtain a bilinear vector; acquiring the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vector, and acquiring a classification result based on the probability distribution condition; thereby realizing the classification and identification of the commodities; according to the method and the device, the attention area image with more detailed information such as details and textures is subjected to feature extraction, and the feature information of the area image is combined, so that the fine feature difference of the commodity is identified by adopting the bilinear vector, and the accuracy of commodity classification identification can be improved.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a product classification and identification device in some embodiments of the present application. This commodity classification recognition device includes: a first extraction module 201, a first generation module 202, a second extraction module 203, a pooling module 204, and an identification module 205.
The first extraction module 201 is configured to extract first feature information of each area image of a product distribution image by using a first convolutional neural network, where the product distribution image includes a plurality of mutually disjoint area images, and each area image corresponds to a product. The commodity distribution image is a distribution image of commodities in the container shot by a fisheye camera in the container. The region image is rectangular or square, each region image corresponds to an image of a commodity, and the plurality of region images are the same in size. When arranging commodity, in order to be convenient for shoot and be convenient for access, a plurality of commodities in this packing cupboard are the rectangular array interval and arrange.
Wherein, a preset first convolution neural network a can be adopted to perform feature extraction on each region image to obtain a corresponding feature fA(X). The first convolutional neural network a is a convolutional neural network commonly used in the prior art for extracting picture feature information.
It is to be appreciated that in some embodiments, the first extraction module 201 is further configured to: acquiring a commodity distribution image, and generating a plurality of calibration frames in the commodity distribution image, wherein each calibration frame comprises a commodity, and the calibration frames are not intersected with each other; and extracting the image of the area surrounded by each calibration frame to obtain a corresponding area image. The calibration frames are rectangular, and any two adjacent calibration frames do not intersect. Then, an image in which each of the calibration frames encloses an area is extracted from the product distribution image with the calibration frame as a boundary, thereby obtaining a plurality of area images.
The first generating module 202 is configured to generate a corresponding attention area map according to the area image and the corresponding first feature information. The attention area map is an image of an area where the feature information is extracted in the area image. Wherein, the attention area graph and the attention area are both square. Specifically, the first generating module 202 includes: a first obtaining unit, configured to input the first feature information into a preset attention area extraction model to obtain location information of an attention area; the second acquisition unit is used for cutting the corresponding region image according to the position information of the attention region so as to acquire an initial attention region map; and the amplifying unit is used for performing up-sampling processing on the initial attention area map so as to acquire the attention area map with the same resolution as that of the corresponding area image.
The predetermined attention area extraction model is composed of at least two convolution layers connected with each other, and may be two or more than two convolution layers. The position information of the attention area is { tx, ty, t1}, where (tx, ty) is the center coordinate of the attention area, the t1 is half the side length of the attention area, and the attention area defaults to a square. The attention area image obtained through cutting is rectangular, a rectangular area with more extracted feature information is cut out from the corresponding area image according to the position information during specific operation, and all the extracted first feature information is extracted from the rectangular area.
Specifically, in some embodiments, the second obtaining unit is configured to: generating a mask M, wherein the size and the shape of the mask M are the same as those of each area image, and each coordinate point of the mask M corresponds to each coordinate point of each area image one to one; acquiring a constraint calculation formula of the coordinate point of the mask M and the position information of the attention area; and cutting out the part of the area image, which is positioned in the attention area, according to the constraint calculation formula to obtain an initial attention area map. Wherein, the constraint calculation formula is M (x, y) = { h (x-tx 1) -h (x-tx 2) } { h (y-ty 1) -h (y-ty 2) }; wherein h (x) = 1/(1 + e)-kx). Where, (tx 1, ty 1) is the coordinate of the top left corner of the attention area obtained from { tx, ty, t1} as the position information, and (tx 2, ty 2) is the coordinate of the bottom right corner of the attention area obtained from { tx, ty, t1} as the position information. By setting the value of k, it can be ensured that the value of mask M (x, y) is 1 only when (x, y) is in the attention area; the M (x, y) is thus used to crop the initial attention area map from the corresponding area image.
The second extraction module 203 is configured to extract second feature information of the attention area map by using a second convolutional neural network;the second extraction module 203 may extract features of each region image by using a preset second convolutional neural network B to obtain corresponding features fB(Y). The second convolutional neural network B is a convolutional neural network commonly used in the prior art for extracting picture feature information.
The pooling module 204 is configured to pool the first feature information and the second feature information to obtain a bilinear vector; the pooling module 204 first stores the first feature information fA(X) and the second characteristic information fBAnd (Y) merging, and then performing pooling operation on the merged feature information by adopting a preset pooling layer to obtain a bilinear feature vector V.
The identification module 205 is configured to obtain a probability distribution of the classification of the commodity corresponding to the region image according to the bilinear vector, and obtain a classification result based on the probability distribution. The bilinear vectors may be preprocessed and then input into a preset probability distribution function to obtain the probability distribution of the classification of the commodity corresponding to the region image. Specifically, the identification module 205 is configured to perform a square root operation and an L2 normalization operation on the bilinear vector in sequence to obtain a target vector; inputting the target vector into a softmax function to obtain the probability distribution condition of the classification of the commodity corresponding to the area image; and obtaining a classification result corresponding to the commodity corresponding to the area image according to the probability distribution condition. Wherein, the probability distribution situation is used for describing the probability that the commodity corresponding to the area image belongs to various categories. For example, for a certain region image, the probability distribution obtained at the end is: p1, P2, P3, P4 and P5, wherein the corresponding categories of P1, P2, P3, P4 and P5 are A1, A2, A3, A4 and A5 respectively. Among these P1, P2, P3, P4, and P5, since P2 is the largest, the classification of the certain region image is a2 type.
As can be seen from the above, in the embodiment of the application, the first feature information of each area image of the commodity distribution image is extracted by using the first convolutional neural network, where the area image includes a plurality of mutually disjoint area images, and each area image corresponds to a commodity; generating a corresponding attention area map according to the area image and the corresponding first characteristic information; extracting second characteristic information of the attention area map by adopting a second convolutional neural network; pooling the first characteristic information and the second characteristic information to obtain a bilinear vector; acquiring the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vector, and acquiring a classification result based on the probability distribution condition; thereby realizing the classification and identification of the commodities; according to the method and the device, the attention area image with more definite identifiable details, textures and other information is subjected to feature extraction, and the features of the area image are combined, so that the tiny feature difference of commodity mutual exclusion is identified by adopting the bilinear vector, and the accuracy of commodity classification identification can be improved.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device 3 according to an embodiment of the present application, where the present application provides an electronic device 3, including: the processor 301 and the memory 302, the processor 301 and the memory 302 being interconnected and communicating with each other via a communication bus 303 and/or other form of connection mechanism (not shown), the memory 302 storing a computer program executable by the processor 301, the processor 301 executing the computer program when the computing device is running to perform the method in any of the alternative implementations of the embodiments described above.
The embodiment of the present application provides a storage medium, and when being executed by a processor, the computer program performs the method in any optional implementation manner of the above embodiment.
The storage medium may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic Memory, a flash Memory, a magnetic disk, or an optical disk.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
Furthermore, the functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A commodity classification and identification method is characterized by comprising the following steps:
extracting first characteristic information of each area image of a commodity distribution image by adopting a first convolution neural network, wherein the commodity distribution image comprises a plurality of mutually-disjoint area images, and each area image corresponds to a commodity;
generating a corresponding attention area map according to the area image and the corresponding first characteristic information;
extracting second characteristic information of the attention area map by adopting a second convolutional neural network;
pooling the first characteristic information and the corresponding second characteristic information to obtain a bilinear vector;
and obtaining the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vector, and obtaining a classification result based on the probability distribution condition.
2. The method for classifying and identifying a product according to claim 1, wherein the generating a corresponding attention area map from the area image and corresponding first feature information includes:
inputting the first characteristic information into a preset attention area extraction model to acquire position information of an attention area;
cutting the corresponding region image according to the position information of the attention region to obtain an initial attention region image;
and performing up-sampling processing on the initial attention area map to acquire an attention area map with the same resolution as that of a corresponding area image.
3. The product classification recognition method according to claim 2, wherein the region image has a rectangular shape, and the attention region has a square shape;
the cutting the corresponding region image according to the position information of the attention region to obtain an initial attention region map includes:
generating a mask M, wherein the size and the shape of the mask M are the same as those of the area image, and each coordinate point of the mask M corresponds to each coordinate point of the area image one to one;
acquiring a constraint calculation formula of the coordinate point of the mask M and the position information of the attention area;
and cutting out the part of the area image, which is positioned in the attention area, according to the constraint calculation formula to obtain an initial attention area map.
4. The method for classifying and identifying merchandise according to claim 2, wherein the predetermined attention area extraction model comprises at least two convolution layers connected in series.
5. The method for classifying and identifying commodities according to claim 1, wherein the obtaining a probability distribution of the classification of the commodities corresponding to the region image from the bilinear vector and obtaining a classification result based on the probability distribution comprises:
sequentially carrying out square root taking operation and L2 normalization operation on the bilinear vectors to obtain target vectors;
inputting the target vector into a softmax function to obtain the probability distribution condition of the classification of the commodity corresponding to the area image;
and obtaining a classification result corresponding to the commodity corresponding to the area image according to the probability distribution condition.
6. The method for classifying and identifying commodities as claimed in claim 1, wherein before extracting the first feature information of each region image of the commodity distribution image using the first convolutional neural network, further comprising:
acquiring a commodity distribution image, and generating a plurality of calibration frames in the commodity distribution image, wherein each calibration frame comprises a commodity, and the calibration frames are not intersected with each other;
and extracting the image of the area surrounded by each calibration frame to obtain a corresponding area image.
7. An article classification and identification device, comprising:
the system comprises a first extraction module, a second extraction module and a third extraction module, wherein the first extraction module is used for extracting first characteristic information of each area image of a commodity distribution image by adopting a first convolution neural network, the commodity distribution image comprises a plurality of mutually-disjoint area images, and each area image corresponds to a commodity;
the first generation module is used for generating a corresponding attention area map according to the area image and corresponding first feature information;
the second extraction module is used for extracting second characteristic information of the attention area map by adopting a second convolutional neural network;
the pooling module is used for pooling the first characteristic information and the second characteristic information corresponding to the first characteristic information to obtain a bilinear vector;
and the identification module is used for acquiring the probability distribution condition of the classification of the commodities corresponding to the region image according to the bilinear vector and acquiring a classification result based on the probability distribution condition.
8. The apparatus according to claim 7, wherein the first generating module comprises:
a first obtaining unit, configured to input the first feature information into a preset attention area extraction model to obtain location information of an attention area;
the second acquisition unit is used for cutting the corresponding region image according to the position information of the attention region so as to acquire an initial attention region map;
and the amplifying unit is used for performing up-sampling processing on the initial attention area map so as to acquire the attention area map with the same resolution as that of the corresponding area image.
9. An electronic device comprising a processor and a memory, the memory storing computer readable instructions that, when executed by the processor, perform the method of any of claims 1-6.
10. A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010458232.7A CN111368942B (en) | 2020-05-27 | 2020-05-27 | Commodity classification identification method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010458232.7A CN111368942B (en) | 2020-05-27 | 2020-05-27 | Commodity classification identification method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111368942A CN111368942A (en) | 2020-07-03 |
| CN111368942B true CN111368942B (en) | 2020-08-25 |
Family
ID=71212300
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010458232.7A Active CN111368942B (en) | 2020-05-27 | 2020-05-27 | Commodity classification identification method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111368942B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20240005701A1 (en) * | 2021-09-23 | 2024-01-04 | Intel Corporation | Methods and apparatus for team classification in sports analysis |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110852351A (en) * | 2019-10-22 | 2020-02-28 | 苏州幻岛信息科技有限公司 | Image-based garbage classification method and device, terminal equipment and storage medium |
| CN111144417A (en) * | 2019-12-27 | 2020-05-12 | 创新奇智(重庆)科技有限公司 | Intelligent container small target detection method and detection system based on teacher student network |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106934397B (en) * | 2017-03-13 | 2020-09-01 | 北京市商汤科技开发有限公司 | Image processing method and device and electronic equipment |
| JP6880494B2 (en) * | 2017-03-27 | 2021-06-02 | 株式会社Ｓｃｒｅｅｎホールディングス | Classification device manufacturing method, image classification method, image classification device, semiconductor inspection device and classification standard module |
| CN110738247B (en) * | 2019-09-30 | 2020-08-25 | 中国科学院大学 | Fine-grained image classification method based on selective sparse sampling |
| CN111079604A (en) * | 2019-12-06 | 2020-04-28 | 重庆市地理信息和遥感应用中心(重庆市测绘产品质量检验测试中心) | Method for quickly detecting tiny target facing large-scale remote sensing image |
| CN111126258B (en) * | 2019-12-23 | 2023-06-23 | 深圳市华尊科技股份有限公司 | Image recognition method and related device |
-
2020
- 2020-05-27 CN CN202010458232.7A patent/CN111368942B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110852351A (en) * | 2019-10-22 | 2020-02-28 | 苏州幻岛信息科技有限公司 | Image-based garbage classification method and device, terminal equipment and storage medium |
| CN111144417A (en) * | 2019-12-27 | 2020-05-12 | 创新奇智(重庆)科技有限公司 | Intelligent container small target detection method and detection system based on teacher student network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111368942A (en) | 2020-07-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111061890B (en) | Method for verifying labeling information, method and device for determining category | |
| EP2662831B1 (en) | Comparing virtual and real images of a shopping planogram | |
| CN109117824B (en) | Commodity management method and device, electronic equipment and storage medium | |
| CN111079632A (en) | Training method and device of text detection model, computer equipment and storage medium | |
| CN106971178A (en) | Pedestrian detection and the method and device recognized again | |
| CN111310706B (en) | Commodity price tag identification method and device, electronic equipment and storage medium | |
| CN109919211A (en) | Commodity recognition method, device, system and computer-readable medium | |
| CN112037267B (en) | Method for generating panoramic graph of commodity placement position based on video target tracking | |
| CN110738236A (en) | Image matching method and device, computer equipment and storage medium | |
| CN110807463B (en) | Image segmentation method and device, computer equipment and storage medium | |
| CN113837151B (en) | Table image processing method and device, computer equipment and readable storage medium | |
| CN112419202A (en) | Wild animal image automatic identification system based on big data and deep learning | |
| CN111368942B (en) | Commodity classification identification method and device, electronic equipment and storage medium | |
| US11861668B2 (en) | Method, device, electronic apparatus and storage medium for generating order | |
| CN107918767A (en) | Object detection method, device, electronic equipment and computer-readable medium | |
| CN114220076A (en) | Multi-target detection method, device and application thereof | |
| Özcan et al. | Ground filtering and DTM generation from DSM data using probabilistic voting and segmentation | |
| CN112949629A (en) | Target detection method and device, computer equipment and readable storage medium | |
| CN111833110A (en) | Customer life cycle positioning method and device, electronic equipment and storage medium | |
| Merrad et al. | A Real-time Mobile Notification System for Inventory Stock out Detection using SIFT and RANSAC. | |
| CN115630660A (en) | Barcode positioning method and device based on convolutional neural network | |
| CN115951880A (en) | EPG page compatible method, system and electronic equipment of IPTV | |
| US20230274227A1 (en) | Retail shelf image processing and inventory tracking system | |
| US20230274226A1 (en) | Retail shelf image processing and inventory tracking system | |
| CN114757822A (en) | Binocular-based human body three-dimensional key point detection method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: Room 501, block a, Haier International Plaza, 939 Zhenwu Road, Jimo Economic Development Zone, Qingdao, Shandong 266200 Patentee after: Qingdao Chuangxin Qizhi Technology Group Co.,Ltd. Address before: Room 04, 02 / F, building 4, software industry base, Yuehai street, Nanshan District, Shenzhen City, Guangdong Province Patentee before: SHENZHEN AINNOVATION TECHNOLOGY Co.,Ltd. |
|
| CP03 | Change of name, title or address | ||
| CP01 | Change in the name or title of a patent holder |
Address after: Room 501, block a, Haier International Plaza, 939 Zhenwu Road, Jimo Economic Development Zone, Qingdao, Shandong 266200 Patentee after: Qingdao Chuangxin Qizhi Technology Group Co.,Ltd. Address before: Room 501, block a, Haier International Plaza, 939 Zhenwu Road, Jimo Economic Development Zone, Qingdao, Shandong 266200 Patentee before: Qingdao Chuangxin Qizhi Technology Group Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder | ||
| CP03 | Change of name, title or address |
Address after: Room 501, block a, Haier International Plaza, 939 Zhenwu Road, Jimo Economic Development Zone, Qingdao, Shandong 266200 Patentee after: Innovation Qizhi Technology Group Co.,Ltd. Country or region after: China Address before: Room 501, block a, Haier International Plaza, 939 Zhenwu Road, Jimo Economic Development Zone, Qingdao, Shandong 266200 Patentee before: Qingdao Chuangxin Qizhi Technology Group Co.,Ltd. Country or region before: China |
|
| CP03 | Change of name, title or address |