KR20210127448A - Product classification system and method for filtering similar images using deep learning - Google Patents

Abstract

Disclosed are a product classification system which can filter and provide a matching image similar to a target image selected by a user by using a classification model learned through deep learning, and a method thereof. To this end, the product classification system comprises a management server for: inputting a target image provided by a user to an analysis model trained with a convolutional neural network (CNN) algorithm to extract a feature map and a feature vector of the target image; extracting a matching image similar to the target image from an image database in which a plurality of images are stored by using the feature map and the feature vector as search conditions; comparing and analyzing the feature vectors of the target image and the matching image to calculate a vector score for the matching image; and classifying the matching images with vector scores matching image filtering ranking values. Accordingly, accurate classification of a product selected by a user is possible, so that images of other products are not included in a result value. In addition, the product classification system provides manufacturers and sellers with an opportunity to easily check at what price their products are sold in open markets or global markets.

Description

Product classification system and method that can filter similar images using deep learning

The present invention relates to a product classification system and method capable of classifying images similar to a target image selected by a user, and more particularly, matching similar to a target image selected by a user using a classification model learned through deep learning It relates to a product classification system and method that can filter and provide images.

With the development of information and communication technology and semiconductor technology, various electronic devices are developing into multimedia devices that provide various multimedia services. For example, electronic devices provide various multimedia services such as a messenger service, a broadcasting service, a wireless Internet service, a camera service, and a music reproduction service.

In addition, the electronic device provides a function for classifying and searching images. Such an electronic device may classify the user's images into a predetermined group by using a preset classification criterion, but there is a problem in that an optimized classification result cannot be provided to the user by using a consistent classification criterion.

In addition, artificial intelligence systems have recently been introduced into the image processing field.

The artificial intelligence (AI) system is a computer system that implements human-level intelligence, and unlike the existing rule-based smart system, the machine learns, judges, and becomes smarter by itself. As the artificial intelligence system is used, the recognition rate is improved and the user's taste can be understood more accurately, and the existing rule-based smart system is gradually being replaced by a deep learning-based artificial intelligence system.

In particular, artificial intelligence technology consists of machine learning (deep learning) and element technologies using machine learning.

Such machine learning is an algorithm technology that categorizes/learns the characteristics of input data by itself, and element technology uses machine learning algorithms such as deep learning to simulate functions such as cognition and judgment of the human brain. It consists of technical fields such as visual understanding, reasoning/prediction, knowledge expression, and motion control.

The linguistic understanding is a technology for recognizing and applying/processing human language/text, and includes natural language processing, machine translation, dialogue system, question answering, voice recognition/synthesis, and the like.

The visual understanding is a technology for recognizing and processing objects like human vision, and includes object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, image improvement, and the like.

The inference prediction is a technique for logically reasoning and predicting by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, recommendation, and the like. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation/classification) and knowledge management (data utilization).

The motion control is a technology for controlling autonomous driving of a vehicle and motion of a robot, and includes motion control (navigation, collision, driving), manipulation control (action control), and the like.

Recently, online product purchases are increasing, and technology to analyze user's personalized tastes is developing. The need for artificial intelligence technology for manufacturers and sellers is being demanded.

Accordingly, a content-based image retrieval technique for searching based on a feature vector of an image as a new method used when comparing and retrieving products in an online shopping mall is being developed.

However, this method has limitations in large-capacity image search because the data capacity is large and the cost for the pre-processing process is large, and due to technical limitations, satisfactory search results are not obtained. In particular, there was a problem in that it was difficult to perform an accurate search when the image was not prepared.

In addition, in order to satisfy the main needs of manufacturers or sellers, if the ranking is specified by the similarity value and the results are sorted in the order of the highest similarity value, even if the similarity is 99%, it is a completely different product ( There was a problem in the case of other brands or fakes).

Republic of Korea Patent Publication No. 10-2020-0002332 (announced on Aug. 8, 2020) Republic of Korea Patent Publication No. 10-2019-0098801 (published on August 23, 2019) Republic of Korea Patent Publication No. 10-2019-0052028 (published on May 15, 2019) Republic of Korea Patent Publication No. 10-2020-0010672 (published on January 31, 2020)

Therefore, a first object of the present invention is a product classification system that can improve classification accuracy by designating a threshold range for a characteristic value and a vector value of a CNN (Convolutional Neural Network) analysis result of an image and detecting only when it falls within the corresponding range. is to provide

In addition, a second object of the present invention is to extract a feature map and a feature vector with a deep learning algorithm for image analysis, extract a matching image based on the feature map and the feature vector, and set the feature vector of the matching image and the supervised learning result The goal is to provide a product classification method that is used to detect identical images by using the vector score for the matching area created by comparing .

In order to achieve the first object of the present invention described above, in an embodiment of the present invention, a target image provided by a user is input to an analysis model learned by a CNN (Convolutional Neural Network)-based algorithm, and a feature map and features of the target image are input. extracting a vector, extracting a matching image similar to the target image from an image database in which a plurality of images are stored using the feature map and the feature vector as a search condition, and comparing and analyzing the feature vector of the target image and the matching image to obtain a matching image It provides a product classification system including a management server for calculating a vector score of and classifying a matching image having a vector score matching the image filtering ranking value.

In addition, in order to achieve the second object of the present invention, in one embodiment of the present invention, the management server receives a target image from a user terminal and extracts a feature map and a feature vector of the target image, the feature vector extraction step, A matching image search step in which the management server extracts a matching image having a feature map and a feature vector matched to the feature map and feature vector of the target image from existing images, and the management server performs the feature map and feature vector of the target image A vector score calculation step of calculating a vector score of the matching image by comparing and analyzing the feature map and the feature vector of the matching image, and the management server checks whether the vector score of the matching image matches the image filtering ranking value, and the image It provides a product classification method including an image classification step of classifying a matching image matching the filtering ranking value as the same-similar image of the target image.

According to the present invention, accurate classification of the product selected by the user is possible, so that images of other products are not included in the result value.

In addition, the present invention provides a manufacturer or seller with an opportunity to easily check at what price their product is sold in an open market or a global market.

In addition, the present invention can provide a search result that searches for the same or similar image as the target image, but excludes images of other brands and images of fakes.

In addition, the present invention can classify a product in circulation into a genuine product and a counterfeit product by comparing and analyzing the photograph of the genuine product and the photograph of the product in circulation, and the manufacturer or the seller can easily check whether there is a product having a design similar to their own product. can

1 is a configuration diagram for explaining a product classification system according to the present invention.
2 is a block diagram illustrating a product classification system according to the present invention.
3 is a flowchart for explaining the operation of the analysis engine according to the present invention.
4 is a schematic diagram for explaining the concept of a vector score calculation and image filtering operation according to the present invention.
5 is a flowchart illustrating a product classification method according to the present invention.

Hereinafter, a product classification system (hereinafter, abbreviated as 'deep learning-based product classification system') that can filter similar images using deep learning according to preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. do.

1 is a block diagram illustrating a product classification system according to the present invention, and FIG. 2 is a block diagram illustrating a product classification system according to the present invention.

1, the deep learning-based product classification system according to the present invention extracts a feature map and a feature vector of a target image with an analysis model learned by a CNN (Convolutional Neural Network)-based algorithm, and , extracting a matching image using the feature map and the feature vector, and comparing and analyzing the feature vector of the target image and the matching image to categorize the same or similar matching image, and optionally the management server 100 ) may further include a user terminal (not shown) that provides the target image.

The user terminal receives a target image or image from a user, and transmits the target image or image to the management server 100, and the management server 100 and a wired/wireless communication network (hereinafter abbreviated as 'communication network') connected through

As an embodiment, the user terminal according to the present invention may transmit the file format of the target image or video selected by the user to the management server 100 as it is without converting the file format.

As another embodiment, the user terminal according to the present invention may format the file format of the target image or video selected by the user into a preset file format and then transmit it to the management server 100 . In this case, the user terminal may format the target image or video to be transmitted to the management server 100 in a low-capacity file format such as MPG or MP4 to reduce the file size.

In this way, the user terminal may convert the format of the target image or video input from the user into a predetermined format and transmit it to the management server 100 .

If necessary, a product classification application or a product classification program may be installed in the user terminal to help access the management server 100 and to convert a file format of a target image or video.

Specifically, the product classification application connects to the management server 100 through a communication network according to a user's request, and transmits a target image or video to the management server 100 .

In addition, when a list of identical images is transmitted from the management server 100 through the communication network, the product classification application outputs the list of identical images to the screen of the user terminal.

In addition, the product classification application may be downloaded through an Android market, an app store, or a web page and installed in the user terminal.

This user terminal is targeted to the management server 100 that provides a deep learning-based product classification service through a wired communication network such as the Internet and a wireless communication network including a number of base stations and base station controllers, mobile communication exchanges and WAP gateways, etc. It means a means for transmitting an image, and a wireless terminal and a wired terminal can be used.

In this case, a smart phone, a tablet PC, a notebook computer, etc. may be used as a wireless terminal, and a desktop computer having an Internet connection means, a smart TV, a wire phone, etc. may be used as a wired terminal.

Meanwhile, the management server 100 categorizes images similar to the target image provided by the user using deep learning, and transmits a list of products similar to or identical to the target image to the user providing the target image.

To this end, the management server 100 inputs a target image to an analysis model learned by a CNN-based algorithm, extracts a feature map and a feature vector of the target image, and uses the feature map and the feature vector as a search condition for a plurality of images. extracts a matching image similar to the target image from the stored image database 122, and compares and analyzes feature vectors of the target image and the matching image to calculate a vector score of the matching image. Here, the vector score may consist of a matrix of numbers between 0 and 1, and the size of the matrix may be determined according to an image analysis algorithm, and the size of the matrix of the feature vector values of the image analysis algorithm matches.

And the management server 100 classifies the matching image having a vector score that matches the image filtering ranking value. Here, the image filtering ranking value is calculated by comparing and analyzing the vector scores of the images that exactly match and the inconsistent images as a result of supervised learning, and input the vector score showing the smallest error into the analysis model trained with the Recurrent Neural Network. mean the average value.

Since the maximum value of this image filtering ranking value is less than 1, even if the vector score of the matching image is close to 1, it may be excluded from the classification of the same-similar image. In other words, according to the present invention, even a matching image having the highest vector score can be classified into a target image and a dissimilar image if the image filtering criterion is exceeded.

In addition, the CNN-based algorithm is any one of FRCNN (Fast Region-based Convolutional Neural Network), Convolutional Neural Network (CNN), RCNN (Region-based Convolutional Neural Network), and SPPnet (Spatial Pyramid Pooling in Deep Convolutional Networks).

Hereinafter, each component of the management server will be described in more detail with reference to the drawings.

1 and 2 , the management server 100 according to the present invention includes a product classification engine 110 .

The product classification engine 110 includes an image preprocessing unit 111, an image conversion unit 112, and an analysis engine 113, and optionally a product option input unit 114, a unit image extraction unit 115, or All of these may be further included.

The image preprocessing unit 111 performs image preprocessing of the target image or image provided by the user, and provides the image preprocessing target image to the image conversion unit 112 or the analysis engine 113 . .

In this case, format conversion or noise removal may be used as image preprocessing. This format conversion is an image pre-processing that facilitates image analysis in the analysis engine 113, and a RAW format may be used.

When the image provided by the user is a moving image, the image preprocessor 111 separates the moving image for each frame and extracts a feature image region included in the frame to generate a target image.

The image conversion unit 112 converts the target image that has passed through the image preprocessor 111 to a predetermined resolution (up scale or down scale), and provides the resolution-converted target image to the analysis engine 113 . . For example, the image converter 112 may convert the resolution of the target image to 640*480, 240*400, or the like.

The product option input unit 114 receives a product search option to be used for product search together with a target image from a user and provides it to the analysis engine 113 unit. When inputted, it is provided to the analysis engine 113 in association with the target image. Here, the product search option is any one or more selected from the group consisting of product name, seller name, manufacturer name, seller contact information, price, size, color, quality, color, and volume of the product.

The product option input unit 114 selects product search keywords through a normalization process such as removing stopwords included in the product search option, and links the product search keyword with the input target image to provide it to the analysis engine 113. can

The unit image extraction unit 115 generates a unit image by extracting only the image region including the characteristics of the analysis target from the target image provided from the image conversion unit 112 , and converts the unit image to the analysis engine 113 . to provide.

The analysis engine 113 inputs the target image provided from the image pre-processing unit 111 or the image conversion unit 112 or the unit image provided from the unit image extraction unit 115 into the analysis model learned by the CNN algorithm to generate a feature map and Extracting a feature vector, extracting a matching image similar to the target image from the image database 122 in which a plurality of images are stored using the feature map and the feature vector as a search condition, and comparing and analyzing the feature vector of the target image and the matching image It provides a function of calculating a vector score of a matching image and classifying a matching image having a vector score matching the image filtering ranking value.

When a target image to which the product search option is linked is received, the analysis engine 113 uses the product search option, the feature map, and the feature vector as a search condition from the image database 122 in which a plurality of images are stored similar to the target image. Matching images can be extracted.

3 is a flowchart for explaining the operation of the analysis engine according to the present invention, and FIG. 4 is a schematic diagram for explaining the concept of the vector score calculation and image filtering operation according to the present invention.

As a specific aspect, the analysis engine 113 according to the present invention may be configured to include a CNN analyzer and a vector score extractor as shown in FIG. 3 .

The CNN analyzer extracts a feature map and a feature vector of the target image by inputting the target image to an analysis model trained with a CNN-based algorithm.

The vector score extractor extracts a matching image similar to the target image from an image database (DB) in which a plurality of images are stored using a feature map and a feature vector of the target image as search conditions, and sets the feature vector of the matching image as the feature of the target image The vector score of the matching image is calculated by comparison with the vector, and the matching image having a vector score matching the image filtering ranking value is classified as the same-similar image.

Here, the feature map and the feature vector are variables that determine whether images are similar, and the vector score is a variable that determines whether image filtering is passed when the feature map and the feature vector are similar patterns.

In addition, since the image filtering ranking value changes according to the type of product, when a target image is selected by the user, the initial value is determined according to the type of the target image, and is changed through self-learning of the analysis engine 113 .

If necessary, the vector score extractor may calculate a vector score of the matching image by applying a weight according to the image matching result of the feature map and the feature vector to the feature vector.

1 and 2 , the management server 100 according to the present invention may further include a big data storage 120 .

The big data storage 120 includes a big data engine 121 and an image DB 122 , and may optionally further include a document DB 123 and a Hadoop Distributed File System (HDFS) 124 .

The big data engine 121 is an integrated storage middleware that accesses each DB to provide query results, searches the image database 122 according to the request of the analysis engine 113, extracts matching images, and then the analysis engine 113. provided as

In addition, when the big data engine 121 provides a feature map, a feature vector, and a feature ID of the target image together with the target image from the analysis engine 113 , it is stored in the image database 122 .

The image DB 122 means a database and file-based storage that can be queried with sql statements such as traditional RDBMS, DBMS, and File DB, and the document DB 123 means a storage space for unstructured document data.

The HDFS 124 is a Hadoop file system, which is a storage of distributed cluster files, and mainly stores FRCNN analysis results, training data, test data, and artificial intelligence analysis data such as image indexes.

5 is a flowchart illustrating a product classification method according to the present invention.

Referring to FIG. 5 , the product classification method according to the present invention includes a feature vector extraction step ( S100 ) of extracting a feature map and a feature vector by receiving a target image from a user terminal, and matching the feature map and feature vector of the target image Matching image search step (S200) of extracting a matching image having the specified feature map and feature vector from existing images, and calculating a vector score of calculating a vector score of the matching image by comparing and analyzing the feature vectors of the target image and the matching image It includes a step (S300), and an image classification step (S400) of filtering the vector score of the matching image with an image filtering ranking value and classifying the matching image that has passed the filtering as the same-similar image.

The product classification method according to the present invention may optionally further include a product image collection step, an analysis model learning step, and a pre-processing step before the feature vector extraction step ( S100 ).

The product image collection step is a step in which the management server 100 collects product images from a web site accessible through a communication network through crawling. After collecting product images through crawling of products in a representative Internet shopping mall site such as 11st, it is stored in the image database 122 .

The analysis model learning step is a step of learning the analysis model by inputting a plurality of product images collected through the product image collection step into the analysis model, respectively. In this case, the analysis model is an analysis model based on a Convolutional Neural Network (CNN) series algorithm, which is one of the deep learning techniques. That is, if each product image is input to a CNN-based algorithm, the qualities of the product image such as shape, color, and pattern can be extracted from each layer of CNN through parameter adjustment.

If necessary, the analysis model can be constructed by stacking CNN-based algorithms in several layers. Such a neural network structure is implemented by using an open structure that has been trained, such as GoogleNet and ResNet, or artificial neural network libraries such as TensorFlow, Torch, and Keras.

As an embodiment, in the pre-processing step according to the present invention, the user terminal may convert the format of the target image input from the user into a predetermined format and transmit it to the management server 100 .

As another embodiment, in the pre-processing step according to the present invention, the management server 100 may convert the format of the target image input from the user terminal into a predetermined format.

In the feature vector extraction step (S100), the management server 100 receives a target image from the user terminal and extracts a feature map and a feature vector of the target image. In this case, the management server 100 may store the feature map and the feature vector of the extracted target image in the image database 122 of the big data storage 120 .

In the feature vector extraction step (S100), the management server 100 may directly receive the target image from the user terminal, or may extract the target image from the video after receiving the video from the user terminal.

The feature vector extracting step (S100) includes an image input process in which the management server 100 receives and stores a target image from a user terminal, an image analysis process in which the target image is converted to a predetermined resolution, and a target whose resolution is converted. It may include a unit image generation process of generating a unit image including a feature image region based on the image, and a feature vector extraction process of extracting a feature map and a feature vector of the unit image.

In the matching image search step (S200), the management server 100 extracts a matching image having a feature map and a feature vector matched to the feature map and feature vector of the target image from existing images stored in the image database 122 do. In this case, the existing images may be stored in the image database 122 in a state in which their feature maps and feature vectors are calculated and linked.

In the feature vector extraction step (S100), the management server 100 receives a product search option from the user terminal, and in the matching image search step, the management server 100 provides a product search option, a feature map, and a feature vector of the target image. A matching image having a product search option, a feature map, and a feature vector matched to can be extracted from existing images.

In the vector score calculation step (S300), the management server 100 compares and analyzes the feature vector of the target image and the feature vector of the matching image to calculate a vector score of the matching image.

In the vector score calculation step ( S300 ), preset weights may be given to the matrix of feature vectors of the target image and the matrix of feature vectors of the matching image according to the product search option, and a vector score may be calculated.

In the image classification step (S400), the management server 100 checks whether the vector score of the matching image matches the image filtering ranking value, and classifies the matching image matching the image filtering ranking value as the same-similar image of the target image. . In this case, the management server 100 may separately manage the matching images classified as identical and similar images to generate the identical similar image list, and transmit the identical similar image list to the user terminal providing the target image.

Although described above with reference to the preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below. You will understand that you can.

100: management server 110: product classification engine
111: image pre-processing unit 112: image conversion unit
113: analysis engine 114: product option input unit
115: unit image extraction unit 120: big data storage
121: big data engine 122: image database
123: document database 124: HDFS

Claims (14)

A target image provided from a user is input to an analysis model trained with a CNN (Convolutional Neural Network)-based algorithm, a feature map and a feature vector of the target image are extracted, and a plurality of images are obtained by using the feature map and the feature vector as a search condition. A matching image similar to the target image is extracted from the stored image database, a vector score of the matching image is calculated by comparing and analyzing feature vectors of the target image and the matching image, and a matching image having a vector score matching the image filtering ranking value. A product classification system including a management server to classify. The method of claim 1, wherein the CNN-based algorithm is
Product classification system, characterized in that any one of FRCNN (Fast Region-based Convolutional Neural Network), CNN, RCNN (Region-based Convolutional Neural Network), and SPPnet (Spatial Pyramid Pooling in Deep Convolutional Networks). The method of claim 1, wherein the image filtering ranking value is
Product classification system, characterized in that it is the average value calculated by inputting the value showing the smallest error by comparing and analyzing the vector scores of the matching image matching the target image and the inconsistent matching image with the target image as a result of supervised learning into an analysis model trained with a recurrent neural network . The method of claim 1, wherein the management server
an image preprocessing unit that performs image preprocessing of a target image provided by a user;
an image conversion unit converting the target image on which the image pre-processing has been performed into a predetermined resolution;
The resolution-converted target image is input to the analysis model trained with a CNN-based algorithm to extract a feature map and a feature vector of the target image, and a plurality of images are stored from an image database using the feature map and feature vector as a search condition. Analysis of extracting a matching image similar to the target image, comparing and analyzing feature vectors of the target image and the matching image to calculate a vector score of the matching image, and classifying matching images having a vector score matching the image filtering ranking value A product classification system, characterized in that it comprises an engine. The method of claim 4, wherein the management server
Further comprising a product option input unit for receiving a product search option to be used for product search together with the target image and providing it to the analysis engine,
and the analysis engine extracts a matching image similar to the target image from an image database in which a plurality of images are stored using the product search option, the feature map, and the feature vector as search conditions. The method of claim 5, wherein the product search option is
Product classification system, characterized in that at least one selected from the group consisting of product name, seller name, manufacturer name, seller contact information, price, size, color, quality, color, and volume of the product. 5. The method of claim 4, wherein the image preprocessor
When the video provided by the user is a video, the video is divided into frames, and a target image is generated by extracting a feature image area included in the frame. 5. The method of claim 4, wherein the analysis engine is
a CNN analyzer for extracting a feature map and a feature vector of the target image by inputting the target image to an analysis model trained with a CNN algorithm; and
Using the feature map and feature vector as a search condition, a matching image similar to the target image is extracted from an image database in which a plurality of images are stored, and a weight according to the image matching result of the feature map and feature vector is applied to the feature vector to obtain a matching image Product classification system, characterized in that it is composed of a vector score extractor that calculates a vector score of and classifies a matching image having a vector score that matches the image filtering ranking value. a feature vector extraction step in which the management server receives a target image from a user terminal and extracts a feature map and a feature vector of the target image;
a matching image search step in which the management server extracts a matching image having a feature map and a feature vector matched to the feature map and feature vector of the target image from existing images;
a vector score calculation step in which the management server compares and analyzes the feature map and feature vector of the target image and the feature map and feature vector of the matching image to calculate a vector score of the matching image; and
Product classification method comprising an image classification step of the management server confirming whether the vector score of the matching image matches the image filtering ranking value, and classifying the matching image matching the image filtering ranking value as the same-similar image of the target image . 10. The method of claim 9, wherein before the feature vector extraction step
and a pre-processing step of converting, by the user terminal, the format of the target image input from the user into a predetermined format and transmitting the format to a management server. 10. The method of claim 9, wherein before the feature vector extraction step
Product classification method, characterized in that the management server further comprises a pre-processing step of converting the format of the target image input from the user terminal into a predetermined format. 10. The method of claim 9, wherein the feature vector extraction step
An image input process in which the management server receives and stores the target image from the user terminal;
An image analysis process of converting the target image to a predetermined resolution;
A unit image generation process of generating a unit image including a feature image area based on the resolution-converted target image, and
and a feature vector extraction process of extracting a feature map and a feature vector of the unit image. 10. The method of claim 9,
In the feature vector extraction step, the management server receives a product search option input from the user terminal, and in the matching image search step, the management server performs a product search option and a feature map of the target image and a product search option and feature map matched to the feature vector. and extracting a matching image having a feature vector from existing images. 14. The method of claim 13,
The vector score calculation step applies a preset weight to the matrix of the feature vector of the target image and the matrix of the feature vector of the matching image according to the product search option, and calculating the vector score.

Images