KR20220076650A - Apparatus and method for searching goods information in retail environment based on goods information image - Google Patents

Abstract

The control method of the product information retrieval device includes the steps of acquiring a product information image photographed by a user terminal, identifying product information from the obtained product information image, and based on the identified product information, the product information image is captured. Storing attribute information in a database, wherein the step of identifying product information includes specifying at least one object included in the product information image using an artificial neural network model, and extracting text data of the specified object and storing in the database includes storing attribute information of the product based on text data.

Description

Apparatus and method for product information search in a retail environment based on product information image {APPARATUS AND METHOD FOR SEARCHING GOODS INFORMATION IN RETAIL ENVIRONMENT BASED ON GOODS INFORMATION IMAGE}

The present invention relates to a product information retrieval device and a control method therefor in an offline retail environment, and more particularly, to an invention that utilizes an image related to product information existing in an offline retail store.

As various online businesses are revitalized, the traditional distribution business is also under pressure to develop differentiated promotion methods. However, many products are still sold in offline stores or online stores. In addition, 91% of global shopping is still done in offline stores, and information on consumers' purchasing journeys is a key piece of information that can confirm what stimuli and information they have encountered in offline stores.

However, it is difficult to collect data obtained through offline stores compared to online data, and despite the fact that it is attracting attention as very valuable data because it is data containing the offline life of consumers, both offline and online stores are It was difficult to understand the purchase journey.

To provide a product information retrieval device and a control method for this, so that personalized services can be utilized throughout the business by collecting and analyzing detailed information about what experiences consumers prefer and react to during the purchase process.

A method for controlling a product information retrieval device according to an aspect includes the steps of obtaining a product information image photographed by a user terminal, identifying product information from the obtained product information image, and a product information image based on the identified product information Storing the property information of the photographed product in a database, wherein the step of identifying product information includes specifying at least one object included in the product information image using an artificial neural network model, and The step of extracting the text data, the step of storing in the database includes the step of storing the attribute information of the product based on the text data.

The artificial neural network model may include a convolutional neural network (CNN) model.

The object may include at least one of a product name, a price, promotional information, a component, and a unique identification code.

The step of identifying the product information further includes the step of receiving location information of a place where the product information image was photographed, and the step of storing the product information in the database includes: store information corresponding to the product information image based on the received location information The control method of the product information retrieval device further comprising the step of specifying and storing store information.

The storing of the attribute information may include storing the attribute information by specifying the extracted object.

The control method of the product information search apparatus may further include performing pre-processing on the product information image before the step of specifying at least one object.

The performing of the preprocessing may include performing at least one of image binarization transformation, noise removal, surrounding background removal, and resolution change.

The control method of the product information retrieval device includes the steps of specifying a product of interest of a user who has photographed a product information image based on attribute information, specifying service information corresponding to the product of interest, and providing the service information to a user terminal may further include.

The step of specifying the product of interest of the user may include specifying the product of interest of the user by using a deep learning-based prediction algorithm.

Deep learning-based prediction algorithms may include Word2Vec and Continuous Bag-of-Words (CBOW) models.

The specifying of the service information may include specifying, as the service information, information on a recommended product having attribute information corresponding to the attribute information of the product of interest.

In a product information retrieval device according to another aspect, the product information retrieval device includes a processor, the processor acquires a product information image photographed by a user terminal, identifies product information from the acquired product information image, and identifies the product information Storing the attribute information of the product for which the product information image is taken based on the database, but identifying the product information, at least one object included in the product information image using a convolutional neural network (CNN) model To specify, to extract the text data of the specified object, and to store in the database, stores the attribute information of the product based on the text data.

The processor may perform pre-processing on the product information image before specifying at least one object.

The processor may specify a product of interest of the user who has photographed the product information image based on the attribute information, specify service information corresponding to the product of interest, and provide the service information to the user terminal.

The processor may specify the product of interest of the user by using a deep learning-based prediction algorithm.

The processor may specify, as service information, information on a recommended product having attribute information corresponding to attribute information of the product of interest.

According to the above-described problem solving means, by collecting information on a consumer's various offline purchase journeys, a personalized service can be utilized throughout the business.

1 is an exemplary diagram of a product information retrieval device and a product information image acquired by the product information retrieval device.
2 is a view showing various objects included in a product information image.
3 is an exemplary diagram for explaining a pre-processing process for a product information image by the product information search apparatus.
4 is an exemplary view of a screen provided by the product information search device.
5 is a conceptual diagram of attribute information stored by a product information search apparatus according to an exemplary embodiment.
6 is a conceptual diagram of vectorized and stored attribute information.
7 is a flowchart of a method for controlling a product information search apparatus according to an exemplary embodiment.

Like reference numerals refer to like elements throughout. This specification does not describe all elements of the embodiments, and general content in the technical field to which the present invention pertains or content that overlaps between the embodiments is omitted. The term 'part, module, member, block' used in this specification may be implemented in software or hardware, and according to embodiments, a plurality of 'part, module, member, block' may be implemented as one component, It is also possible for one 'part, module, member, block' to include a plurality of components.

Throughout the specification, when a part is "connected" to another part, it includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Terms such as first, second, etc. are used to distinguish one component from another, and the component is not limited by the above-mentioned terms.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In each step, the identification code is used for convenience of description, and the identification code does not describe the order of each step, and each step may be performed differently from the specified order unless the specific order is clearly stated in the context. have.

The product information retrieval device may be a user terminal installed with an application that performs a series of processes by photographing a product information image, receives the product information image taken from the user terminal, performs a series of processes, and exchanges data with the user terminal It could be a server that Hereinafter, for convenience of description, an example in which the product information search apparatus is implemented in the form of a user terminal will be described.

The user terminal may be implemented as a computer or a portable terminal. Here, the computer includes, for example, a laptop equipped with a web browser, a desktop, a laptop, a tablet PC, a slate PC, and the like, and the portable terminal has, for example, portability and mobility. As a covered wireless communication device, Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), International Mobile Telecommunication (IMT) -2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, all kinds of handheld based such as smart phone of wireless communication devices and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-devices (HMDs).

The product name or unique identification code is written on the product. Alternatively, a price stack on which product information is written may be located in a portion adjacent to the product.

1 is an exemplary diagram of a product information retrieval device and a product information image acquired by the product information retrieval device.

The product information search apparatus 100 may photograph an adjacent object together with the product. The product information image captured by the product information search apparatus 100 may include a product name, a unique identification code, or other information.

The product information image refers to a photographed image including information for identifying a product or information for distinguishing a specific product from other products. For example, the product information image may include product information. The product information may include information about various objects, such as a product name, price, promotion information, and a unique identification code. That is, the product information image may be an image obtained by photographing a price stack with product information as shown in FIG. , may be an image of a packaging material in which product information is recorded as shown in FIG. 1 ( c ).

The product information search apparatus 100 may identify product information from the photographed product information image.

Specifically, the product information search apparatus 100 may specify at least one object in the product information image.

2 is a view showing various objects included in a product information image.

For example, the product information search apparatus 100 may include a product name (Ob1), a price (Ob2; 'price before discount', 'price after discount', etc.), promotion information (Ob3; '1+1', 'discount', 'Accumulation', 'Giveaway', etc.), and at least one of the unique identification code Ob4 may be specified. In this case, the product information search apparatus 100 may specify an object using a convolutional neural network (CNN) model. As an example, the product information search apparatus 100 is a deep learning R-CNN (" Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik, in CVPR 2014") An object can be specified using an object detection technique based on the model.

Convolution The 'convolution' operation in the neural network model is an operation that extracts image components and can be implemented based on Equation 1 below.

[Equation 1]

By passing the kernel matrix through the digitized image matrix, the result is calculated by calculating the data of the pixel at its location and neighboring pixels, and the data stored as the output matrix has the characteristics of the image.

However, if only the convolution operation is continued, overfitting with lower accuracy occurs from a specific point in time. and dropout can be performed.

Thereafter, the product information search apparatus 100 may extract the position of the object from the product information image using a data classification algorithm (eg, Support Vector Machine (SVM), SoftMax) and may specify the object.

Also, in order to increase the accuracy of the product information image, the product information search apparatus 100 may perform pre-processing on the product information image before specifying an object. The pre-processing may be, for example, image binarization transformation, noise removal, surrounding background removal, and resolution change.

3 is an exemplary diagram for explaining a pre-processing process for a product information image by the product information search apparatus.

Referring to FIG. 3 , as an example, the product information search apparatus 100 may extract a price stack image (a price tag image of FIG. 3 ) from a product information image and convert the price stack image into an image of a uniform size. And, by calculating the coordinates of the objects in the price stack, it is possible to convert the image data for the object into numerical data by numerical data for each coordinate. The product information search apparatus 100 may specify an object based on numerical data of an image according to such coordinates.

Also, the product information search apparatus 100 may identify text in the photographed image and extract text data. In this case, the product information search apparatus 100 may identify the text by reading the entire image area, or may identify the text for each specified object.

4 is an exemplary view of a screen provided by the product information search device.

Referring to FIG. 4 , the product information search apparatus 100 may provide a guide area for displaying product information to be located within a predetermined area to the user. The product information search apparatus 100 may identify the text by reading only the guide area in the product information image. As an embodiment, the product information search apparatus 100 may detect text in an image using an optical character recognition (OCR) method or the like.

The product information search apparatus 100 may train a deep learning model by dividing the case where there is a typo among the recognized text results and the case where there is no typo in the recognized text result and matching the misrecognized product name with the corrected product name in order to accurately detect the text.

Also, the product information search apparatus 100 may acquire location information of a place where the product information image is photographed as meta information. According to an embodiment, information on the shooting time may also be acquired as meta information. The location information may be obtained by extracting shooting coordinates from the product information image, or may be obtained by collecting separate GPS and the like.

The product information search apparatus 100 may store attribute information of a product for which a product information image is captured in a database based on the recognized text data for each object.

5 is a conceptual diagram of attribute information stored by a product information search apparatus according to an exemplary embodiment.

Referring to FIG. 5 , the product information search apparatus 100 may store attribute information (eg, product name, price, promotion information, ingredients, and unique identification code, etc.) of a corresponding product for each product category in a database. . In addition, the product information search apparatus 100 may specify store information corresponding to the product information image based on location information of a place where the product information image is captured, and store the store information as attribute information together.

As an example, the product information search apparatus 100 may vectorize and store product attribute information using a deep learning-based prediction algorithm.

6 is a conceptual diagram of vectorized and stored attribute information.

Specifically, as shown in FIG. 6 , the product information retrieval device 100 utilizes Word2Vec to vectorize ingredients and attribute information for each cosmetic type, and to vectorize the distance between cosmetic ingredients using a CBOW (Continuous Bag-of-Words) model. can Since Word2Vec and CBOW are well-known technologies, detailed descriptions thereof will be omitted. This is to enable a product to be recommended in the future, including properties information calculated on the basis of the user's shooting record and search history, etc. and a component that is close to the vector.

In addition, the product information retrieval apparatus 100 may specify a product of interest of the user who has photographed the product information image, specify service information corresponding to the product of interest, and provide the specified service information to the user.

For example, when a user shoots a product information image of a specific product, another product having the same or similar vector component to the specific product (eg, having a vector value within a preset reference range) as service information to the user may provide.

In addition, the product information search apparatus 100 may provide store information where other products are sold as well as service information (eg, sales information of other cheaper stores, or information on alternative products with the same/similar properties, etc.). .

As described above, the product information retrieval device 100 secures a large amount of learning data with only the product information image, and stores the attribute information by upgrading, so that the service information is specified and provided to the user, thereby providing customized attribute information for each category. It can be filled, and the amount of information that is helpful for users in making a purchase decision can be increased, so that the quality of the service can be improved.

The product information retrieval device 100 may include hardware resources and/or software necessary to implement the technical idea of the present invention, and does not necessarily mean one physical component or one device. not. That is, the photographing-based information providing system 100 may mean a logical combination of hardware and/or software provided to implement the technical idea of the present invention, and if necessary, installed in devices spaced apart from each other It may be implemented as a set of logical configurations for implementing the technical idea of the present invention by performing a function.

In addition, the product information search apparatus 100 uses an algorithm for controlling the operation of components in the product information search apparatus 100 or a memory for storing a database for a program or application that reproduces the algorithm, and the database stored in the memory. Thus, it may include a processor that performs the above-described operation. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

Hereinafter, a method of controlling the product information search apparatus 100 will be described with reference to FIG. 7 . 7 is a flowchart of a method for controlling a product information search apparatus according to an exemplary embodiment.

First, the product information search apparatus 100 according to an embodiment acquires a product information image captured by the user terminal ( 1100 ).

Next, the product information search apparatus 100 according to an embodiment identifies product information from the acquired product information image ( 1200 ).

Here, the operation 1200 of identifying product information includes an operation of specifying at least one object included in the product information image using a convolutional neural network (CNN) model, and extracting text data of the specified object. It may include an action to

The object may include at least one of a product name, a price, promotional information, a component, and a unique identification code.

For the operation of specifying an object, as described above in relation to deep learning R-CNN, pooling, and dropout, the redundant description will be omitted.

The operation of extracting the text data has been described above in relation to the optical character reading method, and thus a redundant description will be omitted.

Also, the operation 1200 of identifying product information may further include performing pre-processing on the product information image before specifying at least one object. The operation of preprocessing the product information image may include at least one of image binarization transformation, noise removal, surrounding background removal, and resolution change on the product information image.

And, the operation 1200 of identifying the product information may further include an operation of receiving location information of a place where the product information image was photographed.

Next, the product information search apparatus 100 according to an embodiment stores attribute information of a product for which a product information image is captured based on the identified product information in the database ( 1300 ).

The operation 1300 of storing the attribute information of the product on which the product information image is photographed in the database includes an operation of specifying the extracted object and storing the attribute information, and store information corresponding to the product information image based on the location information. It may include an operation of specifying and storing store information.

On the other hand, the product information search apparatus 100 goes further than storing attribute information, it is also possible to provide service information to the user based on the stored attribute information.

The product information search apparatus 100 according to another embodiment may specify the product of interest of the user who has photographed the product information image by using a database for stored attribute information.

The operation of specifying the product of interest of the user may include the operation of specifying the product of interest of the user by using a deep learning-based prediction algorithm. In this case, the deep learning-based prediction algorithm may include a Word2Vec and a continuous bag-of-words (CBOW) model, and a redundant description thereof will be omitted.

Next, the product information search apparatus 100 according to another embodiment may specify service information corresponding to the specified user's interest product.

The operation of specifying the service information may include an operation of specifying, as service information, information on a recommended product having attribute information corresponding to the attribute information of the product of interest.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create a program module to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

The computer-readable recording medium includes any type of recording medium in which instructions readable by the computer are stored. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like.

The disclosed embodiments have been described with reference to the accompanying drawings as described above. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be practiced in other forms than the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

100: product information search device

Claims (13)

A method for controlling a product information retrieval device, comprising:
acquiring a product information image taken from a user terminal;
identifying product information from the obtained product information image; and
Comprising the step of storing the attribute information of the product for which the product information image is photographed in a database based on the identified product information,
The step of identifying the product information,
Using an artificial neural network model to specify at least one object included in the product information image, and
Extracting the text data of the specified object,
The step of storing in the database,
A control method of a product information search apparatus comprising the step of storing attribute information of the product based on the text data. The method of claim 1,
The artificial neural network model is a control method of a product information search device including a convolutional neural network (CNN) model. The method of claim 1,
The object is a control method of a product information retrieval device including at least one of a product name, price, promotion information, a component, and a unique identification code. The method of claim 1,
The step of identifying the product information,
Further comprising the step of receiving the location information of the place where the product information image was photographed,
The step of storing in the database is,
Specifying store information corresponding to the product information image based on the received location information, and storing the store information. The method of claim 1,
The step of storing the attribute information includes:
A control method of a product information retrieval device comprising the step of specifying the extracted object and storing attribute information. The method of claim 1,
Prior to the step of specifying the at least one object,
The control method of the product information search apparatus further comprising the step of performing pre-processing on the product information image. 7. The method of claim 6,
Performing the pre-processing step,
A control method of a product information retrieval apparatus for performing at least one of image binarization conversion, noise removal, surrounding background removal, and resolution change. The method of claim 1,
specifying a product of interest of a user who has photographed the product information image based on the attribute information;
specifying service information corresponding to the product of interest; and
The control method of the product information search apparatus further comprising the step of providing the service information to the user terminal. 9. The method of claim 8,
The step of specifying the product of interest of the user,
A control method of a product information search device comprising the step of specifying the product of interest of the user by using a deep learning-based prediction algorithm. 10. The method of claim 9,
The deep learning-based prediction algorithm is a control method of a product information search device including a Word2Vec and a continuous bag-of-words (CBOW) model. The method of claim 1,
The step of specifying the service information,
and specifying, as the service information, information on a recommended product having attribute information corresponding to the attribute information of the product of interest, as the service information. A product information retrieval device comprising:
The product information retrieval device includes a processor,
The processor is
Acquire the product information image taken from the user terminal,
Identify product information in the obtained product information image,
Storing the attribute information of the product for which the product information image is taken based on the identified product information in a database,
To identify the product information,
Using a Convolutional Neural Network (CNN) model to specify at least one object included in the product information image, extract the text data of the specified object,
Storing in the database is
A product information retrieval device for storing attribute information of the product based on the text data. acquiring a product information image taken from a user terminal;
identifying product information from the obtained product information image; and
Comprising the step of storing the attribute information of the product for which the product information image is photographed in a database based on the identified product information,
The step of identifying the product information,
Using a convolutional neural network (CNN) model to specify at least one object included in the product information image, comprising the step of extracting the text data of the specified object,
The step of storing in the database is,
A computer-readable recording medium having recorded thereon a method comprising the step of storing attribute information of the product based on the text data.

Images