KR20200010672A - Smart merchandise searching method and system using deep learning - Google Patents

Abstract

According to the present invention, a smart product search method using deep learning collects images to be learned by using an image crawler, learns the collected images by using a graphic processing unit (GPU), and based on the learned results, recognizes keywords and/or images input by a use to provide highly relevant information on a user screen. The present invention relates to a product searching method of the deep learning manner, which performs learning in the deep learning manner by using a variety of product images, and with the same, displays more accurate search results than any conventional method. In particular, the present invention can more quickly and accurately search a product by comparing a search result image with an image used in learning through image feature extraction by the deep learning manner, thereby preventing the lack of data which is a problem of the conventional product search method, and a search error generated by incorrect matching of the image and the keyword.

Description

Smart merchandise searching method and system using deep learning}

The present invention relates to a smart product searching method in an online shopping mall, and more particularly, to a method for quickly and accurately searching for a product by matching a keyword of a product with an image in an online shopping mall database.

Currently, the method used when comparing and searching for products in an online shopping mall is generally a method of showing them to users based on attribute information based on information of a product input as text. This method is a keyword-based image retrieval technology, and if there is a lack of knowledge about products, it is necessary to directly check the products of all classifications, and it is difficult to accurately match the keywords of the image during the search, which leads to many difficulties in product search. As a new method, a content-based retrieval technique for searching based on a feature vector of an image may be used. This method has a limitation in retrieving large images because of the large data size and the cost for the preprocessing process, and the technical limitations do not provide satisfactory search results. In particular, it is difficult to perform an accurate search when the image is not prepared.

On the other hand, a technique for performing content-based image retrieval using image attribute information by mixing the aforementioned two methods has also been studied. However, even a system supporting such a search method is difficult to properly interpret various user queries and display results, and there may be a problem when integrating or exchanging with other systems.

Accordingly, the present invention is to solve this problem, and to provide a method for quickly and accurately searching for a desired product in an online shopping mall only by inputting a limited or incorrect keyword or image.

In order to solve the above problems, in the present invention, first, by using an image crawler to collect the image to learn, using the graphic processing unit (Graphic Processing Unit; GPU) to learn, based on the learned results By recognizing the keyword and / or image input by the user, it provides a highly relevant information on the user screen.

The present invention relates to a product learning method of the deep learning method to learn in a deep learning method using a variety of product images, and to search for a product using this shows more accurate search results than any conventional method. In particular, the present invention provides a search error that occurs due to a lack of data, an incorrect matching of an image and a keyword, which is a problem of the conventional product search method, and the image and search result image used for learning through image feature extraction by the deep learning method. The invention makes it possible to search products faster and more accurately by comparing them.

1 illustrates a schematic diagram of an online shopping mall product image learner and an image recognizer.
Fig. 2 shows a schematic diagram of a convolutional neural network and a traditional neural network in the form of a rare relationship.

The development of the online shopping environment allows consumers to compare different products in one place. However, there is a limit that a considerable amount of main product information posted on the online shopping mall cannot be reflected in a text-based retrieval system that can be recognized by a computer because it is an image form. These limitations can generally be improved by recognizing keywords in the form of images using existing machine learning techniques and OCR (Optical Character Recognition) techniques. However, the existing OCR technology has a problem of low recognition rate when there are many non-letter pictures in the image and the font size is small. In order to solve the limitations of the existing technologies, a text recognition model in the product catalog in the form of an image was designed by modifying a single shot multibox detector (SSD), which is one of the deep learning based object recognition models. However, it took considerable time and money to build the data to train it, because SSD models that follow the supervised methodology require direct labeling of each training data.

Unlike this prior art, the present invention is based on machine learning and deep learning by artificial intelligence.

Artificial Intelligence (AI) is the ability of a machine to imitate intelligent human behavior. AI can be divided into general AI and applied AI. Applied AI, also called vertical AI or narrow AI, refers to a "smart" system that specializes in specific needs, such as stock trading or customized advertising. General AI, also called Strong AI or Full AI, refers to systems and devices that can handle all the tasks that humans can handle. These are similar to the droids found in science fiction films. When the general public thinks about the future, much of it is related to general AI.

Machine learning, a sub-domain of AI, contributes to much of the advances in the field of AI, including image recognition and natural language processing. Machine learning basically analyzes data using algorithms, learns through analysis, and makes judgments or predictions based on what is learned. Ultimately, the goal is to learn how to perform tasks by 'learning' the computer itself through large amounts of data and algorithms, rather than coding specific guidelines for decision criteria directly in software.

Deep learning, on the other hand, is a cutting-edge technology inspired by the human brain structure, which uses artificial neural networks to process data in a manner similar to human brain cells. A massive amount of data is introduced into the neural network, which "trains" the system to accurately classify the data. The emergence of today's supercomputers and big data has paved the way for deep learning. Another algorithm created by early machine learning researchers, the artificial neural network, inspired the biological properties of the human brain, especially neuronal connections. However, unlike the brain, where any neuron in physical proximity can be interconnected, artificial neural networks have a constant layer connection and data propagation direction. For example, if you cut an image into many tiles and enter it into the first layer of the neural network, the neurons repeat the process of passing the data to the next layer until the final output is produced on the last layer. Each neuron is assigned a weight that indicates the accuracy of the input based on the task being performed, and then the final output is determined by summing all the weights.

Deep learning is a form of artificial intelligence developed from an artificial neural network as described above, and uses data input and output layers similar to neurons in the brain to learn data. But even basic neural networks require huge amounts of computation, so neural networks are likely to produce a number of incorrect answers during the 'learning' process. However, the greater the time and amount of learning, the greater the probability of getting close to the correct answer.

In deep learning, computer models learn to classify directly from images, text, or sound. Also, since most deep learning methods use a neural network architecture, the deep learning model is often called a deep neural network. The term 'deep' refers to the number of hidden layers that make up a neural network. Traditional neural networks have only two or three hidden layers, but up to 150 deep networks. The most popular type of deep neural network is the Convolutional Neural Network (CNN), which extracts features by taking a composite product of the input data. Using images as an example, a well-trained deep learning model can automatically identify objects in an image even if they have never seen the same image before.

Synthetic multiplication neural network (CNN) is a type of multilayer perceptrons designed to use minimal pretreatment. CNN consists of one or several convolutional layers and a common artificial neural network layer on top of it, further utilizing weights and pooling layers. This structure allows the CNN to fully utilize the input data of the two-dimensional structure. Compared to other deep learning architectures, CNN performs well in both video and audio. CNNs can also be trained through standard reverse propagation. CNN is easier to train than other feedforward neural network techniques and has the advantage of using fewer parameters.

Machine learning can be a direct learning process that extracts knowledge from data, but it is common to learn at the stage of "data-feature-knowledge" after intermediate feature extraction. For example, in order to recognize an object in a picture, a characteristic line or characteristic color distribution is first extracted from a pixel value, and then, based on this, the object is determined. This intermediate representation is called a feature map, and the performance of machine learning depends largely on how good features are extracted. In particular, a synthetic multiplying neural network that extracts and learns features in multiple steps is useful for image recognition.

In addition, machine learning or data extraction using the compound multiplication neural network can be further improved through the following techniques.

1. Data Augmentation

Machine learning requires a large amount of data, because learning about various cases can lead to general estimates. If the amount of data is small, the estimation is only good for the learned data, which leads to the problem of overfitting, which affects the performance of the AI model. Problems that may arise when the number of data is insufficient may be compensated by a method of flipping an image or rotating the image.

2. Convolution Filter

The composite product neural network convolves the numerical values of the input image included in the corresponding size with a predetermined filter size and outputs the result. Expressed as an equation, when the filter is W, the convolution result can be expressed in the form of Wx + b (b is bias). This is linear modeling. The deeper the AI, the more sophisticated classifiers can be designed through the combination of linear and nonlinear functions.

The size of the image through convolution depends on the size of the filter and the stride in which the filter moves within the image. If the input image size is N × N and the filter size is F × F, the size of the image output through this filter is (NF) / stride +1, indicating that the feature map of the next layer is affected by the size of the filter. It can be seen that. Basically, as the filter passes, the size of the input image becomes smaller. In order to prevent this, it is necessary to amplify the input image.

3. max pooling

Maximum pooling refers to sampling convolutional feature maps. For example, assuming a feature map of 4x4 and a filter of 2x2, this method extracts the largest value among the values inside the 2x2 filter to create a 2x2 result layer. According to this technique, a feature map can be extracted from only the strongest features from the input image.

4. softmax layer and one-hot encoding

Softmax does the job of changing the result of the score to a value between 0 and 1, which adds up to 1. An exp is taken for each score and then divided by a normalization constant to calculate the sum to 1.

In the last stage of the neural network, vector values for N classes corresponding to the corresponding class are calculated. Each vector corresponding to the number of classes has an arbitrary value. As the result passes through the softmax layer, each result is changed to a vector between 0 and 1. Thereafter, the one-hot encoding converts the highest value to one and the remaining values to zero, so that the most characteristic result can be extracted.

5. Cost function and gradient descent algorithm

In order to learn the weight value of artificial intelligence, we define the objective function. The objective function depends on the purpose, but in general, the cross entropy method is used. Known ground truth is defined using the objective function using a feature vector over the softmax layer. In this case, when the gradient descent method is used, the direction of the objective function is minimized.

When using gradient descent algorithms, providing too many learning epochs tends to produce neural networks that are tailored to specific inputs and actual information. If a question is asked, it is solved correctly, but if another problem comes out, the result may not be as expected. To prevent this, the problem of overfitting can be solved by adding a dropout layer, input noise, and weight noise.

Accordingly, in the method of the present invention, the machine learning of the synthetic product neural network may be performed through the following steps:

i) Receive training sample data and neural network data

ii) prepare neural network data as graphical data

iii) for each input sample

Calculation result output

Calculate the error for the expected output

-Determination of objective function

-Reset convolution filter

Repeating the above

iv) Enter next sample-repeat

In the present invention, through the image learning by deep learning, based on the result of learning as many and accurate keywords as possible to match the product image, the user's input keyword and / or the image that is closest to the image will be provided to the user screen Can be.

On the other hand, the present invention also provides a system for carrying out such a method,

Keyword and / or image input by the user;

An image determination unit for selecting an image closest to the input keyword and / or the image;

A result output including the selected image

Including,

The image determination unit

Distributed file storage system,

Deep Neural Network based single object image recognition learning unit

DNN model profile generation unit,

A DNN profile extraction and model generation unit, and

-Product image recognition unit

It also provides a system comprising a.

Furthermore, the present invention also provides a method and an apparatus, characterized in that learning is automatically performed when a keyword is not pre-learned.

In addition, in another preferred embodiment of the present invention, not only provide a result matching the keyword and / or image input by the user, but also provide a method and apparatus that can be customized to provide other information suitable for the user. Can be.

Claims (4)

As a product search method in an online shopping mall,
Collecting images to learn using an image crawler,
Learning the collected image using a graphic processing unit (GPU),
Recognizing keywords and / or images input by the user based on the learned results and providing relevant information to the user screen
Product search method comprising a. The method of claim 1,
The providing of highly relevant information to a user screen may include recommending content of interest to the user. The method of claim 1,
The learning may include automatically learning a keyword and / or an image that has not been pre-learned. As a product search system in an online shopping mall,
Keyword and / or image input by the user;
An image determining unit for selecting an image closest to the input keyword and / or the image; And
A result output including the selected image
Including,
The image determination unit
Distributed file storage system,
Deep Neural Network based single object image recognition learning unit
DNN model profile generation unit,
A DNN profile extraction and model generation unit, and
-Product image recognition unit
Search system comprising a.

Images