CN111126384A - Commodity classification system and method based on feature fusion - Google Patents
Commodity classification system and method based on feature fusion Download PDFInfo
- Publication number
- CN111126384A CN111126384A CN201911273063.3A CN201911273063A CN111126384A CN 111126384 A CN111126384 A CN 111126384A CN 201911273063 A CN201911273063 A CN 201911273063A CN 111126384 A CN111126384 A CN 111126384A
- Authority
- CN
- China
- Prior art keywords
- image
- commodity
- sample
- matching
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 34
- 230000004927 fusion Effects 0.000 title claims abstract description 18
- 238000000605 extraction Methods 0.000 claims abstract description 48
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims abstract description 43
- 230000009466 transformation Effects 0.000 claims abstract description 43
- 239000000284 extract Substances 0.000 claims description 15
- 238000013527 convolutional neural network Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/02—Affine transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Biomedical Technology (AREA)
- Evolutionary Biology (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a commodity classification system and method based on feature fusion, wherein the system comprises: the image input module is used for inputting a commodity image to be detected; the image shooting angle estimation module is used for estimating the shooting angle of the commodity image; the image affine transformation module is used for carrying out affine transformation on the commodity image according to the estimated shooting angle; the image global feature extraction module is used for carrying out global feature extraction on the commodity image after affine transformation; the first image matching module is used for carrying out image matching on the global characteristic graph and various graphs prestored in the sample graph library to obtain a first matching result; the image local feature extraction module is used for extracting local features of the commodity image subjected to affine transformation; and the second image matching module is used for performing local feature matching on the local feature map and various maps which are preliminarily matched and have relevance with the commodity image to obtain a classification result of the commodity image.
Description
Technical Field
The invention relates to the technical field of commodity identification and detection, in particular to a commodity classification system and a classification method based on feature fusion.
Background
In the field of unmanned selling, the commodities need to be automatically identified and classified. In recent years, with the development of computer vision recognition technology, automatic commodity identification detection methods based on vision recognition technology are beginning to be applied in large quantities. However, the number of commodities in a retail scene is large, the types of commodities are various, the shooting scene of commodity pictures is complex, the commodity images shot by the same commodity in different scenes and different angles have great difference, the commodity emission on the goods shelf is dense, the volume of a single commodity is usually small, similar commodities on the goods shelf are often placed together, so that the similarity between the commodities and the classes of the commodities is high, the error rate of classification and identification of the commodities with complex scenes by the conventional automatic commodity identification and detection method is high, and the commodities cannot be classified finely.
Disclosure of Invention
The invention aims to provide a commodity classification system and method based on feature fusion to solve the technical problems.
In order to achieve the purpose, the invention adopts the following technical scheme:
provided is a commodity classification system based on feature fusion, comprising:
the image input module is used for providing a user with input of a commodity image to be detected;
the image shooting angle pre-estimation module is connected with the image input module and used for pre-estimating the shooting angle of the commodity image according to a reference object in the commodity image;
the image affine transformation module is respectively connected with the image input module and the image shooting angle estimation module and is used for carrying out affine transformation on the commodity image according to the estimated shooting angle so as to correct the angle of the commodity image in the commodity image;
the image global feature extraction module is connected with the image affine transformation module and is used for carrying out image global feature extraction on the commodity image after affine transformation to obtain a global feature map related to the commodity image;
the first image matching module is connected with the image global feature extraction module and a sample map library and is used for carrying out image matching on the global feature map and various sample maps prestored in the sample map library one by one, preliminarily matching the sample maps which are relevant to the commodity image according to the similarity between the global feature map and the sample maps and storing the sample maps;
the image local feature extraction module is connected with the image affine transformation module and is used for carrying out image local feature extraction on the commodity image after affine transformation to obtain a local feature map corresponding to the commodity image;
the second image matching module is respectively connected with the image local feature extraction module and the first image matching module and is used for further performing image local feature matching on the local feature map and each sample map obtained by preliminary matching, and further matching the sample maps which are related to the commodity image in each sample map obtained by preliminary matching according to the similarity between the local feature map and the local features of each matched sample map and storing the sample maps;
and the commodity category judging module is connected with the second image matching module and used for outputting the commodity category corresponding to the sample image obtained by final matching as the commodity category corresponding to the commodity.
As a preferable aspect of the present invention, the article classification system further includes:
the first characteristic difference region extraction module is respectively connected with the image local characteristic extraction module and the second image matching module and is used for identifying and extracting and storing a first characteristic difference region of each sample image obtained by further matching on the local characteristic image;
the second characteristic difference region extraction module is respectively connected with the image local characteristic extraction module and the second image matching module and is used for identifying, extracting and storing second characteristic difference regions of the local characteristic images and the sample images obtained through further matching;
the third image matching module is respectively connected with the first characteristic difference area extraction module and the second characteristic difference area extraction module, and is used for performing area image matching on the first characteristic difference area and each second characteristic difference area one by one, and matching according to the area similarity of the first characteristic difference area and each second characteristic difference area to obtain the sample image associated with the commodity image and storing the sample image;
the commodity category judging module is connected with the third image matching module and used for outputting the commodity category corresponding to the sample image obtained by matching of the third image matching module as the commodity category corresponding to the commodity in the commodity image.
As a preferable scheme of the invention, a convolutional neural network is adopted to extract the global features and the local features of the commodity image.
The invention also provides a commodity classification method based on feature fusion, which is realized by applying the commodity classification system and comprises the following steps:
step S1, the commodity classification system acquires the commodity image to be detected;
step S2, the commodity classification system estimates the shooting angle of the commodity image according to the reference object in the commodity image;
step S3, the commodity classification system carries out affine transformation on the commodity image according to the shooting angle estimated in the step S2;
step S4, the commodity classification system performs image global feature extraction on the commodity image after affine transformation to obtain the global feature map associated with the commodity image;
step S5, the commodity classification system carries out image matching on the global feature map and each sample map prestored in the sample map library one by one, and according to the similarity between the global feature map and each sample map, the sample maps which are related to the commodity image are obtained through preliminary matching;
step S6, the commodity classification system extracts image local features of the commodity image after affine transformation, and extracts a local feature map corresponding to the commodity image;
step S7, the product classification system further performs image local feature matching on the local feature map obtained in step S6 and each of the sample maps matched in step S5, and further matches out the sample map having a relationship with the product image in each of the sample maps preliminarily matched according to a similarity between the local feature map and a local feature of each of the sample maps;
step S8, the product classification system outputs the product category corresponding to the sample chart obtained by matching in step S7 as the product category corresponding to the product in the product image.
The invention also provides a commodity classification method based on feature fusion, which is realized by applying the commodity classification system and comprises the following steps:
step L1, the commodity classification system acquires the commodity image to be detected;
l2, the commodity classification system pre-estimates the shooting angle of the commodity image according to the reference object in the commodity image;
step L3, the commodity classification system carries out affine transformation on the commodity image according to the shooting angle estimated in the step L2;
step L4, the commodity classification system performs image global feature extraction on the commodity image after affine transformation to obtain the global feature map associated with the commodity image;
step L5, the commodity classification system performs image matching on the global feature map and each sample map prestored in the sample map library one by one, and performs preliminary matching according to the similarity between the global feature map and each sample map to obtain the sample map associated with the commodity image;
step L6, the commodity classification system performs image local feature extraction on the commodity image after affine transformation, and extracts a local feature map corresponding to the commodity image;
a step L7 of performing further image local feature matching on the local feature map and each of the sample maps matched in the step L5 by the product classification system, and further matching sample maps having a relationship with the product image in each of the sample maps preliminarily matched according to a similarity between the local feature map and a local feature of each of the sample maps;
a step L8 of identifying and extracting a first feature difference region of each sample map matched with the step L7 on the local feature map and storing the first feature difference region, and identifying and extracting a second feature difference region of each sample map matched with the step L7 on the local feature map and storing the second feature difference region;
step L9, the product classification system performs area image matching on the first feature difference area and each of the second feature difference areas, and obtains the sample drawing associated with the product image by matching according to the similarity between the first feature difference area and each of the second feature difference areas;
and a step L10 of outputting, by the product classification system, the product category corresponding to the sample chart obtained by the matching in the step L9 as the product category corresponding to the product in the product image.
The invention has the beneficial effects that:
1. the method comprises the steps of firstly estimating the picture shooting angle through a fixed reference object in the commodity image, and then carrying out affine transformation on the commodity image to align the commodity image, so that the influence of the commodity image on subsequent commodity image identification and classification due to the picture shooting quality problem is solved;
2. according to the method, global features of the commodity image are extracted, the commodity image is subjected to image matching with the sample image, the commodity category in the commodity image is preliminarily judged, then local features of the commodity image are extracted, the extracted local features are further subjected to image matching with various images which are obtained through preliminary judgment and have relevance with the commodity image, and the accuracy of commodity identification is improved;
3. according to the invention, the interesting region in the local features corresponding to the commodity image is extracted, and then the interesting region is continuously subjected to image matching with the matched sample image, so that the precision degree of commodity classification is favorably improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a schematic structural diagram of a commodity classification system based on feature fusion according to a first embodiment of the present invention;
FIG. 2 is a diagram of the method steps of a method for classifying commodities based on feature fusion according to a first embodiment of the present invention;
fig. 3 is a schematic structural diagram of a commodity classification system based on feature fusion according to a second embodiment of the present invention;
fig. 4 is a method step diagram of a commodity classification method based on feature fusion according to the second embodiment of the present invention.
Detailed Description
The technical scheme of the invention is further explained by the specific implementation mode in combination with the attached drawings.
Wherein the showings are for the purpose of illustration only and are shown by way of illustration only and not in actual form, and are not to be construed as limiting the present patent; to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The same or similar reference numerals in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. are used for indicating the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, it is only for convenience of description and simplification of description, but it is not indicated or implied that the referred device or element must have a specific orientation, be constructed in a specific orientation and be operated, and therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and are not to be construed as limitations of the present patent, and the specific meanings of the terms may be understood by those skilled in the art according to specific situations.
In the description of the present invention, unless otherwise explicitly specified or limited, the term "connected" or the like, if appearing to indicate a connection relationship between the components, is to be understood broadly, for example, as being fixed or detachable or integral; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or may be connected through one or more other components or may be in an interactive relationship with one another. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
Example one
An embodiment of the present invention provides a system for classifying commodities based on feature fusion, with reference to fig. 1, including:
the image input module 1 is used for providing a user with input of a commodity image to be detected;
the image shooting angle estimation module 2 is connected with the image input module 1 and used for estimating the shooting angle of the commodity image according to reference objects in the commodity image, such as reference objects such as a shelf and the like, and the method for estimating the shooting angle of the commodity image is the prior art, so the estimation process of the shooting angle is not explained herein;
the image affine transformation module 3 is respectively connected with the image input module 1 and the image shooting angle estimation module 2 and is used for carrying out affine transformation on the commodity image according to the estimated shooting angle so as to correct the angle of the commodity image in the commodity image; the method for carrying out affine transformation on the commodity image according to the estimated shooting angle is also the prior art, so the specific process of carrying out affine transformation on the commodity image is not explained here;
the image global feature extraction module 4 is connected with the image affine transformation module 3 and is used for carrying out image global feature extraction on the commodity image after affine transformation to obtain a global feature map related to the commodity image; the invention preferably extracts the global image features of the commodity image through the conventional convolutional neural network, and the method for extracting the global image features through the convolutional neural network is the prior art, so the process of extracting the global features is not explained herein;
the first image matching module 5 is connected with the image global feature extraction module 4 and the same image library 100, and is used for performing image matching on the global feature map and various images prestored in the same image library 100 one by one, calculating the similarity between the global feature map and the various images, and preliminarily matching and storing the sample map which is associated with the commodity image according to the similarity calculation result; the method for calculating the similarity between the global feature map and the sample map is the prior art, so the specific process of calculating the similarity is not described here. The number of the sample images which are preliminarily matched and have similarity with the commodity image can be one or more. One sample image corresponds to one commodity category, so that a plurality of sample images which are preliminarily matched possibly correspond to a plurality of commodity categories, so that a preliminary matching result possibly indicates that commodities in the commodity image possibly correspond to the plurality of commodity categories, and in order to realize accurate classification of the commodities in the commodity image, the commodity classification system further comprises:
the image local feature extraction module 6 is connected with the image affine transformation module 3 and is used for performing image local feature extraction on the commodity image after affine transformation to obtain a local feature map corresponding to the commodity image;
the second image matching module 7 is respectively connected with the image local feature extraction module 6 and the first image matching module 5, and is used for further performing image local feature matching on the local feature map and the various images obtained by preliminary matching, and further matching sample maps which are associated with the commodity images in the various images obtained by preliminary matching according to the similarity between the local feature map and the local features of the various matched images and storing the sample maps;
and the commodity type judging module 8 is connected with the second matching image module 7 and is used for outputting the commodity type corresponding to the finally matched sample image as the corresponding commodity type in the commodity image.
In the above technical solution, it should be emphasized that, in the preliminary image matching process, matching objects of the global feature maps corresponding to the commodity images are all the sample maps in the sample map library. And the matching objects of the local characteristic graphs corresponding to the commodity images are the preliminarily matched various patterns.
Referring to fig. 3, an embodiment of the present invention further provides a method for classifying a commodity based on feature fusion, which is implemented by applying the commodity classification system provided in the first embodiment, and the method includes the following steps:
step S1, the commodity classification system acquires a commodity image to be detected;
step S2, the commodity classification system pre-estimates the shooting angle of the commodity image according to the reference object in the commodity image;
step S3, the commodity classification system carries out affine transformation on the commodity image according to the shooting angle estimated in the step S2;
step S4, the commodity classification system extracts the global image feature of the commodity image after affine transformation to obtain a global feature map associated with the commodity image;
step S5, the commodity classification system carries out image matching on the global feature map and various samples prestored in a sample map library one by one, and according to the similarity between the global feature map and various samples, a sample map which is related to the commodity image is obtained through preliminary matching;
step S6, the commodity classification system extracts the local image features of the commodity image after affine transformation, and extracts the local feature map corresponding to the commodity image;
step S7, the commodity classification system carries out further image local feature matching on the local feature map obtained in the step S6 and the various maps matched in the step S5, and further matches out a sample map which is related to the commodity image in the various primarily matched sample maps according to the similarity between the local feature map and the local features of the sample maps;
in step S8, the product classification system outputs the product type corresponding to the sample chart obtained by matching in step S7 as a product type corresponding to the product in the product image.
Example two
The difference between the second embodiment and the first embodiment is that, referring to fig. 2, the product classification system provided by the second embodiment further includes:
the first characteristic difference region extraction module 9 is respectively connected with the image local characteristic extraction module 6 and the second image matching module 7, and is used for identifying and extracting and storing a first characteristic difference region (interested region) of each image obtained by further matching on the local characteristic image;
a second characteristic difference region extraction module 10, respectively connected to the image local characteristic extraction module 6 and the second image matching module 7, for identifying and extracting and storing a second characteristic difference region (region of interest) between each sample image obtained by further matching and the local characteristic image;
the third image matching module 11 is respectively connected with the first characteristic difference area 9 and the second characteristic difference area 10, and is used for performing area image matching on the first characteristic difference area and each second characteristic difference area one by one, and matching according to the area similarity between the first characteristic difference area and each second characteristic difference area to obtain a sample image which is most related to the commodity image and storing the sample image;
and the commodity type judging module 8 is connected with the third image matching module 11 and is used for outputting the commodity type corresponding to the sample image obtained by matching the third image matching module 11 as the commodity type corresponding to the commodity in the commodity image.
In the above technical solution, the method for identifying and extracting the first feature difference region between the local feature map and each sample map and the method for identifying and extracting the second feature difference region between the local feature map and each sample map are both existing feature identification and extraction methods, so the specific identification and extraction process thereof is not described herein.
In addition, the similarity matching between the first feature difference region and the second feature difference region is also the existing matching method, and the specific matching process is not described here.
In addition, the commodity types corresponding to the various graphs are identified and confirmed in advance, that is, the commodity types corresponding to the commodities on the commodity images can be obtained only by matching the graphs which are most relevant to the commodity images, so that the commodity classification efficiency is improved.
Referring to fig. 4, the present invention further provides a method for classifying commodities based on feature fusion, which is implemented by applying the commodity classification system provided in the second embodiment, and specifically includes the following steps:
l1, the commodity classification system acquires a commodity image to be detected;
l2, the commodity classification system pre-estimates the shooting angle of the commodity image according to the reference object in the commodity image;
step L3, the commodity classification system carries out affine transformation on the commodity image according to the shooting angle estimated in the step L2;
l4, the commodity classification system extracts the global image features of the commodity image after affine transformation to obtain a global feature map associated with the commodity image;
l5, the commodity classification system carries out image matching on the global feature map and various samples prestored in the sample map library one by one, and according to the similarity between the global feature map and various samples, a sample map which is related to the commodity image is obtained through preliminary matching;
l6, the commodity classification system extracts the local features of the images of the commodities after affine transformation, and extracts the local feature maps corresponding to the commodity images;
step L7, the commodity classification system further matches the local feature map with the local features of the images matched in step L5, and further matches a sample map with relevance to the commodity image in the preliminarily matched sample maps according to the similarity between the local feature map and the local features of the sample maps;
l8, the merchandise classification system identifies and extracts the first feature difference region of the local feature map matched with the various maps obtained in the L7, stores the first feature difference region and
identifying and extracting a second feature difference area between each sample image obtained by matching in the step L7 and the local feature image, and storing the second feature difference area;
step L9, the commodity classification system performs area image matching on the first characteristic difference area and each second characteristic difference area, and according to the similarity between the first characteristic difference area and each second characteristic difference area, a sample diagram having relevance with the commodity image is obtained through matching;
at step L10, the product classification system outputs the product type corresponding to the sample chart obtained by the matching at step L9 as the product type corresponding to the product in the product image.
In step L8, the method for the product classification system to identify and extract the first feature difference region and the second feature difference region is the prior art, so the specific identification and extraction process thereof is not described here.
In step L9, the process of matching the similarity between the first feature difference region and the second feature difference region by the product classification system is also the prior art, so the process of matching the similarity is not described here. In addition, in step L9, the system outputs the sample image having the highest similarity to the product image, and the product type corresponding to the sample image having the highest similarity as the product type corresponding to the product in the product image.
It should be understood that the above-described embodiments are merely preferred embodiments of the invention and the technical principles applied thereto. It will be understood by those skilled in the art that various modifications, equivalents, changes, and the like can be made to the present invention. However, such variations are within the scope of the invention as long as they do not depart from the spirit of the invention. In addition, certain terms used in the specification and claims of the present application are not limiting, but are used merely for convenience of description.
Claims (5)
1. A commodity classification system based on feature fusion is characterized by comprising:
the image input module is used for providing a user with input of a commodity image to be detected;
the image shooting angle pre-estimation module is connected with the image input module and used for pre-estimating the shooting angle of the commodity image according to a reference object in the commodity image;
the image affine transformation module is respectively connected with the image input module and the image shooting angle estimation module and is used for carrying out affine transformation on the commodity image according to the estimated shooting angle so as to correct the angle of the commodity image in the commodity image;
the image global feature extraction module is connected with the image affine transformation module and is used for carrying out image global feature extraction on the commodity image after affine transformation to obtain a global feature map related to the commodity image;
the first image matching module is connected with the image global feature extraction module and a sample map library and is used for carrying out image matching on the global feature map and various sample maps prestored in the sample map library one by one, preliminarily matching the sample maps which are relevant to the commodity image according to the similarity between the global feature map and the sample maps and storing the sample maps;
the image local feature extraction module is connected with the image affine transformation module and is used for carrying out image local feature extraction on the commodity image after affine transformation to obtain a local feature map corresponding to the commodity image;
the second image matching module is respectively connected with the image local feature extraction module and the first image matching module and is used for further performing image local feature matching on the local feature map and each sample map obtained by preliminary matching, and further matching the sample maps which are related to the commodity image in each sample map obtained by preliminary matching according to the similarity between the local feature map and the local features of each matched sample map and storing the sample maps;
and the commodity category judging module is connected with the second image matching module and used for outputting the commodity category corresponding to the sample image obtained by final matching as the commodity category corresponding to the commodity.
2. The article classification system of claim 1, further comprising:
the first characteristic difference region extraction module is respectively connected with the image local characteristic extraction module and the second image matching module and is used for identifying and extracting and storing a first characteristic difference region of each sample image obtained by further matching on the local characteristic image;
the second characteristic difference region extraction module is respectively connected with the image local characteristic extraction module and the second image matching module and is used for identifying, extracting and storing second characteristic difference regions of the local characteristic images and the sample images obtained through further matching;
the third image matching module is respectively connected with the first characteristic difference area extraction module and the second characteristic difference area extraction module, and is used for performing area image matching on the first characteristic difference area and each second characteristic difference area one by one, and matching according to the area similarity of the first characteristic difference area and each second characteristic difference area to obtain the sample image associated with the commodity image and storing the sample image;
the commodity category judging module is connected with the third image matching module and used for outputting the commodity category corresponding to the sample image obtained by matching of the third image matching module as the commodity category corresponding to the commodity in the commodity image.
3. The commodity classification system according to claim 1, wherein a convolutional neural network is used to perform image global feature extraction and image local feature extraction on the commodity image.
4. A commodity classification method based on feature fusion, which is implemented by applying the commodity classification system as in any one of claims 1 to 3, comprising the steps of:
step S1, the commodity classification system acquires the commodity image to be detected;
step S2, the commodity classification system pre-estimates the shooting angle of the commodity image according to the reference object in the commodity image;
step S3, the commodity classification system carries out affine transformation on the commodity image according to the shooting angle estimated in the step S2;
step S4, the commodity classification system performs image global feature extraction on the commodity image after affine transformation to obtain the global feature map associated with the commodity image;
step S5, the commodity classification system carries out image matching on the global feature map and each sample map prestored in the sample map library one by one, and according to the similarity between the global feature map and each sample map, the sample maps which are related to the commodity image are obtained through preliminary matching;
step S6, the commodity classification system extracts image local features of the commodity image after affine transformation, and extracts a local feature map corresponding to the commodity image;
step S7, the product classification system further performs image local feature matching on the local feature map obtained in step S6 and each of the sample maps matched in step S5, and further matches out the sample map having a relationship with the product image in each of the sample maps preliminarily matched according to a similarity between the local feature map and a local feature of each of the sample maps;
step S8, the product classification system outputs the product category corresponding to the sample chart obtained by matching in step S7 as the product category corresponding to the product in the product image.
5. A commodity classification method based on feature fusion, which is realized by applying the commodity classification system according to claim 1 or 2, and is characterized by comprising the following steps:
step L1, the commodity classification system acquires the commodity image to be detected;
l2, the commodity classification system pre-estimates the shooting angle of the commodity image according to the reference object in the commodity image;
step L3, the commodity classification system carries out affine transformation on the commodity image according to the shooting angle estimated in the step L2;
step L4, the commodity classification system performs image global feature extraction on the commodity image after affine transformation to obtain the global feature map associated with the commodity image;
step L5, the commodity classification system performs image matching on the global feature map and each sample map prestored in the sample map library one by one, and performs preliminary matching according to the similarity between the global feature map and each sample map to obtain the sample map associated with the commodity image;
step L6, the commodity classification system performs image local feature extraction on the commodity image after affine transformation, and extracts a local feature map corresponding to the commodity image;
a step L7 of performing further image local feature matching on the local feature map and each of the sample maps matched in the step L5 by the product classification system, and further matching sample maps having a relationship with the product image in each of the sample maps preliminarily matched according to a similarity between the local feature map and a local feature of each of the sample maps;
a step L8 of identifying and extracting a first feature difference region of each sample map matched with the step L7 on the local feature map and storing the first feature difference region, and identifying and extracting a second feature difference region of each sample map matched with the step L7 on the local feature map and storing the second feature difference region;
step L9, the product classification system performs area image matching on the first feature difference area and each of the second feature difference areas, and obtains the sample drawing associated with the product image by matching according to the similarity between the first feature difference area and each of the second feature difference areas;
and a step L10 of outputting, by the product classification system, the product category corresponding to the sample chart obtained by the matching in the step L9 as the product category corresponding to the product in the product image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911273063.3A CN111126384A (en) | 2019-12-12 | 2019-12-12 | Commodity classification system and method based on feature fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911273063.3A CN111126384A (en) | 2019-12-12 | 2019-12-12 | Commodity classification system and method based on feature fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111126384A true CN111126384A (en) | 2020-05-08 |
Family
ID=70499892
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911273063.3A Pending CN111126384A (en) | 2019-12-12 | 2019-12-12 | Commodity classification system and method based on feature fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111126384A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111767959A (en) * | 2020-06-30 | 2020-10-13 | 创新奇智(广州)科技有限公司 | Method and device for classifying pile fibers |
| CN113837825A (en) * | 2021-08-16 | 2021-12-24 | 上海漫集信息科技有限公司 | Commodity classification method and system |
Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101276419A (en) * | 2007-02-19 | 2008-10-01 | 精工爱普生株式会社 | Category classification apparatus, category classification method, and storage medium storing a program |
| CN102799850A (en) * | 2012-06-30 | 2012-11-28 | 北京百度网讯科技有限公司 | Bar code recognition method and device |
| CN103617420A (en) * | 2013-11-27 | 2014-03-05 | 上海电机学院 | Commodity fast recognition method and system based on image feature matching |
| CN104281679A (en) * | 2014-09-30 | 2015-01-14 | 东软集团股份有限公司 | Goods classification method and goods classification device both based on image features |
| CN105009170A (en) * | 2012-12-28 | 2015-10-28 | 日本电气株式会社 | Object identification device, method, and storage medium |
| US20160180509A1 (en) * | 2014-12-17 | 2016-06-23 | Casio Computer Co., Ltd. | Commodity identification device and commodity recognition navigation method |
| CN106021575A (en) * | 2016-05-31 | 2016-10-12 | 北京奇艺世纪科技有限公司 | Retrieval method and device for same commodities in video |
| CN106169064A (en) * | 2016-06-27 | 2016-11-30 | 张梦石 | The image-recognizing method of a kind of reality enhancing system and system |
| CN106570510A (en) * | 2016-11-07 | 2017-04-19 | 南京航空航天大学 | Supermarket commodity identification method |
| CN106599010A (en) * | 2015-10-14 | 2017-04-26 | 魏立江 | Picture code searching method |
| CN106933816A (en) * | 2015-12-29 | 2017-07-07 | 北京大唐高鸿数据网络技术有限公司 | Across camera lens object retrieval system and method based on global characteristics and local feature |
| CN108228761A (en) * | 2017-12-21 | 2018-06-29 | 深圳市商汤科技有限公司 | The customized image search method in support area and device, equipment, medium |
| CN108319961A (en) * | 2018-01-23 | 2018-07-24 | 西南科技大学 | A kind of image ROI rapid detection methods based on local feature region |
| US20180239989A1 (en) * | 2017-02-20 | 2018-08-23 | Alibaba Group Holding Limited | Type Prediction Method, Apparatus and Electronic Device for Recognizing an Object in an Image |
| CN108734185A (en) * | 2017-04-18 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Image verification method and apparatus |
| CN108920643A (en) * | 2018-06-26 | 2018-11-30 | 大连理工大学 | Weight the fine granularity image retrieval algorithm of multiple features fusion |
| CN109117879A (en) * | 2018-08-03 | 2019-01-01 | 南京旷云科技有限公司 | Image classification method, apparatus and system |
| CN109359666A (en) * | 2018-09-07 | 2019-02-19 | 佳都新太科技股份有限公司 | A kind of model recognizing method and processing terminal based on multiple features fusion neural network |
| WO2019062017A1 (en) * | 2017-09-27 | 2019-04-04 | 缤果可为（北京）科技有限公司 | Method, device, and self-service checkout counter for performing product recognition on the basis of neural network |
| CN109815770A (en) * | 2019-01-31 | 2019-05-28 | 北京旷视科技有限公司 | Two-dimentional code detection method, apparatus and system |
| CN110119455A (en) * | 2019-04-23 | 2019-08-13 | 西安理工大学 | A kind of image classification search method based on convolution depth confidence network |
| CN110414541A (en) * | 2018-04-26 | 2019-11-05 | 京东方科技集团股份有限公司 | The method, equipment and computer readable storage medium of object for identification |
-
2019
- 2019-12-12 CN CN201911273063.3A patent/CN111126384A/en active Pending
Patent Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101276419A (en) * | 2007-02-19 | 2008-10-01 | 精工爱普生株式会社 | Category classification apparatus, category classification method, and storage medium storing a program |
| CN102799850A (en) * | 2012-06-30 | 2012-11-28 | 北京百度网讯科技有限公司 | Bar code recognition method and device |
| CN105009170A (en) * | 2012-12-28 | 2015-10-28 | 日本电气株式会社 | Object identification device, method, and storage medium |
| CN103617420A (en) * | 2013-11-27 | 2014-03-05 | 上海电机学院 | Commodity fast recognition method and system based on image feature matching |
| CN104281679A (en) * | 2014-09-30 | 2015-01-14 | 东软集团股份有限公司 | Goods classification method and goods classification device both based on image features |
| US20160180509A1 (en) * | 2014-12-17 | 2016-06-23 | Casio Computer Co., Ltd. | Commodity identification device and commodity recognition navigation method |
| CN106599010A (en) * | 2015-10-14 | 2017-04-26 | 魏立江 | Picture code searching method |
| CN106933816A (en) * | 2015-12-29 | 2017-07-07 | 北京大唐高鸿数据网络技术有限公司 | Across camera lens object retrieval system and method based on global characteristics and local feature |
| CN106021575A (en) * | 2016-05-31 | 2016-10-12 | 北京奇艺世纪科技有限公司 | Retrieval method and device for same commodities in video |
| CN106169064A (en) * | 2016-06-27 | 2016-11-30 | 张梦石 | The image-recognizing method of a kind of reality enhancing system and system |
| CN106570510A (en) * | 2016-11-07 | 2017-04-19 | 南京航空航天大学 | Supermarket commodity identification method |
| US20180239989A1 (en) * | 2017-02-20 | 2018-08-23 | Alibaba Group Holding Limited | Type Prediction Method, Apparatus and Electronic Device for Recognizing an Object in an Image |
| CN108734185A (en) * | 2017-04-18 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Image verification method and apparatus |
| WO2019062017A1 (en) * | 2017-09-27 | 2019-04-04 | 缤果可为（北京）科技有限公司 | Method, device, and self-service checkout counter for performing product recognition on the basis of neural network |
| CN108228761A (en) * | 2017-12-21 | 2018-06-29 | 深圳市商汤科技有限公司 | The customized image search method in support area and device, equipment, medium |
| CN108319961A (en) * | 2018-01-23 | 2018-07-24 | 西南科技大学 | A kind of image ROI rapid detection methods based on local feature region |
| CN110414541A (en) * | 2018-04-26 | 2019-11-05 | 京东方科技集团股份有限公司 | The method, equipment and computer readable storage medium of object for identification |
| CN108920643A (en) * | 2018-06-26 | 2018-11-30 | 大连理工大学 | Weight the fine granularity image retrieval algorithm of multiple features fusion |
| CN109117879A (en) * | 2018-08-03 | 2019-01-01 | 南京旷云科技有限公司 | Image classification method, apparatus and system |
| CN109359666A (en) * | 2018-09-07 | 2019-02-19 | 佳都新太科技股份有限公司 | A kind of model recognizing method and processing terminal based on multiple features fusion neural network |
| CN109815770A (en) * | 2019-01-31 | 2019-05-28 | 北京旷视科技有限公司 | Two-dimentional code detection method, apparatus and system |
| CN110119455A (en) * | 2019-04-23 | 2019-08-13 | 西安理工大学 | A kind of image classification search method based on convolution depth confidence network |
Non-Patent Citations (2)
| Title |
|---|
| 周东尧;伍岳庆;姚宇;: "基于全局特征和尺度不变特征转换特征融合的医学图像检索", no. 04 * |
| 段娜;王磊;: "全局及其个性化区域特征的图像检索", no. 2 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111767959A (en) * | 2020-06-30 | 2020-10-13 | 创新奇智(广州)科技有限公司 | Method and device for classifying pile fibers |
| CN111767959B (en) * | 2020-06-30 | 2023-10-31 | 创新奇智(广州)科技有限公司 | Plush fiber classifying method and device |
| CN113837825A (en) * | 2021-08-16 | 2021-12-24 | 上海漫集信息科技有限公司 | Commodity classification method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11823429B2 (en) | Method, system and device for difference automatic calibration in cross modal target detection | |
| CN108229468B (en) | Vehicle appearance feature recognition and vehicle retrieval method and device, storage medium and electronic equipment | |
| CN108229322B (en) | Video-based face recognition method and device, electronic equipment and storage medium | |
| EP3910608B1 (en) | Article identification method and system, and electronic device | |
| US9111348B2 (en) | Computer-based method and system of dynamic category object recognition | |
| US9141886B2 (en) | Method for the automated extraction of a planogram from images of shelving | |
| CN108734185B (en) | Image verification method and device | |
| CN112016401A (en) | Cross-modal-based pedestrian re-identification method and device | |
| WO2020073601A1 (en) | Goods recognition method, goods recognition apparatus, and storage medium | |
| CN111061890A (en) | Method for verifying labeling information, method and device for determining category | |
| JP2023015989A (en) | Item identification and tracking system | |
| CN113962274B (en) | Abnormity identification method and device, electronic equipment and storage medium | |
| WO2020134102A1 (en) | Article recognition method and device, vending system, and storage medium | |
| CN110569789B (en) | Commodity combined sku identification method and device | |
| WO2023024443A1 (en) | Data matching method and apparatus, and electronic device, storage medium and program product | |
| CN111126384A (en) | Commodity classification system and method based on feature fusion | |
| Liu et al. | A cloud infrastructure for target detection and tracking using audio and video fusion | |
| CN109743497B (en) | Data set acquisition method and system and electronic device | |
| Achakir et al. | An automated AI-based solution for out-of-stock detection in retail environments | |
| CN111242094A (en) | Commodity identification method, intelligent container and intelligent container system | |
| CN110689005A (en) | Commodity identification method based on deep learning fusion position and shape information | |
| CN111008210B (en) | Commodity identification method, commodity identification device, codec and storage device | |
| CN112365324A (en) | Commodity picture detection method suitable for E-commerce platform | |
| CN115953635B (en) | Sorting method, AR (augmented reality) glasses and system for multi-class targets | |
| CN115937548A (en) | Commodity matching method based on bimodal feature fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |