KR102508420B1 - Device for searching goods information in offline retail environment using purchase receipt image and control method thereof - Google Patents

Abstract

A control method of a product information retrieval apparatus includes acquiring a purchase receipt image captured by a user terminal, identifying purchase information in the obtained purchase receipt image using an artificial neural network model, and using the identified purchase information as a purchase receipt. The method includes specifying a product of interest of the user whose image is captured, specifying service information corresponding to the specified product of interest, and providing the service information to a user terminal.

Description

Product information retrieval device in offline retail environment using purchase receipt image and control method therefor

The present invention relates to a product information retrieval device in an offline retail environment and a control method therefor, and particularly to utilizing an image of a product purchase receipt in an offline retail store.

As various online businesses are active, traditional distribution businesses are also under pressure to develop differentiated promotion methods. However, many products are still sold in offline stores or online stores. In addition, 91% of shopping worldwide is still done in offline stores, and information about the consumer's purchase journey is the main information that can confirm what stimulation and information they have encountered in offline stores.

However, data obtained through offline stores is difficult to collect compared to online data, and although it is attracting attention as very valuable data because it contains actual consumers' offline lives, both offline stores and online stores provide It was difficult to understand the purchasing journey.

It is intended to provide a product information retrieval device and a control method for this, which enables personalized services to be utilized throughout the business by collecting and analyzing detailed information on what experiences consumers prefer and react to during the purchasing process.

A control method of a product information retrieval apparatus according to an aspect includes acquiring a purchase receipt image captured by a user terminal, identifying purchase information from the obtained purchase receipt image using an artificial neural network model, and assigning information to the identified purchase information. Based on the purchase receipt image, specifying the product of interest of the user, specifying service information corresponding to the specified product of interest, and providing the service information to the user terminal.

The step of identifying the purchase information includes specifying at least one object included in the purchase receipt image using an artificial neural network model, and extracting text data of the specified object, and specifying a product of interest. may include specifying the product of interest of the user whose purchase receipt image was captured based on the extracted text data.

The artificial neural network model may include a Convolutional Neural Network (CNN) model.

The object may include at least one of store information, product name, price, promotion information, quantity information, purchase time information, buyer information, and a unique identification code.

The identifying purchase information may further include post-processing the extracted text data, wherein the post-processing may include post-processing the text data based on at least one of character unit information and text context information of the purchase receipt image. processing may be included.

The control method of the product information retrieval apparatus may further include performing pre-processing on the purchase receipt image before identifying the purchase information.

The step of specifying the product of interest is acquired through the purchase receipt image. Receiving user information matching the purchase information of the user from a separate personal information server, selecting at least one product having attribute information most similar to that of the product on which the purchase receipt image was photographed among a plurality of products stored in the database. It may include specifying as a product of interest, and matching and storing user information with the product of interest.

Specifying service information may include specifying, as service information, information about a recommended product having attribute information corresponding to attribute information of a product of interest.

Product information search device according to another aspect, the product information search device includes a processor, the processor acquires a purchase receipt image photographed by a user terminal, and identifies purchase information from the obtained purchase receipt image by using an artificial neural network model. Based on the identified purchase information, the product of interest of the user whose purchase receipt image is captured is specified, service information corresponding to the specified product of interest is specified, and the service information is provided to the user terminal.

Identifying the purchase information includes specifying at least one object included in the purchase receipt image using an artificial neural network model and extracting text data of the specified object, and specifying the product of interest includes the extracted text. It may include specifying the product of interest of the user whose purchase receipt image was captured based on the data.

A computer-readable recording medium on which a method according to another aspect is recorded includes acquiring a purchase receipt image photographed by a user terminal, identifying purchase information from the obtained purchase receipt image by using an artificial neural network model, and identifying purchase information using an artificial neural network model. The method includes specifying a product of interest of the user whose purchase receipt image is captured based on purchase information, specifying service information corresponding to the specified product of interest, and providing the service information to a user terminal.

The step of identifying the purchase information includes specifying at least one object included in the purchase receipt image using an artificial neural network model, and extracting text data of the specified object, and specifying a product of interest. may include specifying the product of interest of the user whose purchase receipt image was captured based on the extracted text data.

According to the above-described problem solving means, by collecting information on various purchase journeys of consumers offline, personalized services can be utilized throughout the business.

1 is an exemplary view of a product information search device and a product information image acquired by the product information search device.
2 is a diagram showing various objects included in a product information image.
3 is an exemplary diagram for explaining a pre-processing process for a product information image by a product information search device.
4 is an exemplary view of a screen provided by a product information search device.
5 is an exemplary diagram of a purchase receipt image.
6 is a diagram illustrating various examples of service information provided by a product search device.
7 is a flowchart of a control method of a product information search apparatus according to an embodiment.

Like reference numbers designate like elements throughout the specification. This specification does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the present invention belongs is omitted. The term 'unit, module, member, or block' used in the specification may be implemented as software or hardware, and according to embodiments, a plurality of 'units, modules, members, or blocks' may be implemented as one component, It is also possible that one 'part, module, member, block' includes a plurality of components.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of being directly connected but also the case of being indirectly connected, and indirect connection includes being connected through a wireless communication network. do.

In addition, when a certain component is said to "include", this means that it may further include other components without excluding other components unless otherwise stated.

Terms such as first and second are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

Expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In each step, the identification code is used for convenience of description, and the identification code does not explain the order of each step, and each step may be performed in a different order from the specified order unless a specific order is clearly described in context. there is.

The product information search device may be a user terminal installed with an application that takes a product information image and performs a series of processes, receives a product information image taken from the user terminal, performs a series of processes, and exchanges data with the user terminal. It can also be a server that Hereinafter, for convenience of description, an example in which a product information search device is implemented as a user terminal will be described.

The user terminal may be implemented as a computer or a portable terminal. Here, the computer includes, for example, a laptop, a desktop, a laptop, a tablet PC, a slate PC, etc. equipped with a web browser, and a portable terminal, for example, has portability and mobility. As a guaranteed wireless communication device, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) -2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone (Smart Phone) based on all types of handhelds It may include a wireless communication device and a wearable device such as a watch, ring, bracelet, anklet, necklace, glasses, contact lens, or head-mounted-device (HMD).

A product name or unique identification code is written on the product. Alternatively, a price tag with product information written thereon may be located adjacent to the product.

1 is an exemplary view of a product information search device and a product information image acquired by the product information search device.

The product information retrieval apparatus 100 may photograph a product and an adjacent object together. The product information image captured by the product information search device 100 may include a product name, a unique identification code, or other information.

The product information image refers to a photographed image including information for identifying a product or information for distinguishing a specific product from other products. For example, the product information image may include product information. Product information may include information on various objects such as product name, price, promotion information, and unique identification code. That is, the product information image may be an image of a price tag with product information written thereon as shown in FIG. 1 (a), or an image of a product with product information written on the surface as shown in FIG. 1 (b) , It may be an image of a packaging material in which product information is written, as shown in (c) of FIG. 1 .

The product information search device 100 may identify product information from a captured product information image.

Specifically, the product information search apparatus 100 may specify at least one object in the product information image.

2 is a diagram showing various objects included in a product information image.

For example, the product information retrieval apparatus 100 may display a product name (Ob1), a price (Ob2; 'price before discount', 'price after discount', etc.), promotion information (Ob3; '1+1', etc.) in the product information image. At least one of 'discount', 'accumulation', 'gift presentation', etc.), and a unique identification code Ob4 may be specified. At this time, the product information retrieval device 100 may specify an object using a convolutional neural network (CNN) model. As an example, the product information retrieval device 100 may use a deep learning R-CNN (" Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik, in CVPR 2014") An object can be specified by object detection technology based on the model.

The 'convolution' operation in the convolutional neural network model is an operation that extracts components of an image and can be implemented based on Equation 1 below.

[Equation 1]

The result is derived by passing the kernel matrix through the digitized image matrix to calculate the data of the pixel at its own location and the pixels adjacent to the periphery, and the data stored as the output matrix has the characteristics of the image.

However, if only the convolution operation is continued, overfitting with low accuracy occurs from a certain point in time. In order to prevent overfitting, the product information retrieval apparatus 100 may pool with large values from adjacent values of the output matrix. Yes, and dropout can be performed.

Thereafter, the product information search apparatus 100 may extract the location of an object from the product information image using a data classification algorithm (eg, Support Vector Machine (SVM), SoftMax) and may specify the object.

In addition, the product information search apparatus 100 may perform pre-processing on the product information image before specifying the object in order to increase the accuracy of the product information image. Pre-processing may be, for example, performing image binarization conversion, noise removal, surrounding background removal, resolution change, and the like.

3 is an exemplary diagram for explaining a pre-processing process for a product information image by a product information search device.

Referring to FIG. 3 , as an example, the product information search apparatus 100 may extract a price stack image (the price tag image of FIG. 3 ) from a product information image and convert the price stack image into a uniformly sized image. In addition, by calculating the coordinates of the objects in the price stack, the image data of the object can be converted into numerical data using numerical data for each coordinate. The product information search apparatus 100 may specify an object based on numerical data of an image according to these coordinates.

Also, the product information search apparatus 100 may identify text in a photographed image and extract text data. At this time, the product information search apparatus 100 may identify text by reading the entire image area or may identify text for each specified object.

As an embodiment, the product information search apparatus 100 may detect text in a product information image using an optical character recognition (OCR) method or the like.

In order to accurately detect text, the product information retrieval apparatus 100 may learn a deep learning model by dividing a case with and without a typo in the recognized text result and then matching the misrecognized product name with the corrected product name.

Also, the product information search apparatus 100 may obtain location information of a place where the product information image was photographed as meta information. Depending on the embodiment, information on the shooting time may also be acquired as meta information. Location information may be obtained by extracting shooting coordinates from a product information image, or may be obtained by collecting a separate GPS or the like.

The product information retrieval apparatus 100 may store attribute information of a product in which a product information image is captured based on text data for each recognized object in a database.

4 is a conceptual diagram of attribute information stored by a product information search device according to an exemplary embodiment.

Referring to FIG. 4 , the product information retrieval apparatus 100 may store attribute information (eg, product name, price, promotion information, ingredients, and unique identification code) of a corresponding product for each product category in a database. . In addition, the product information search apparatus 100 may specify store information corresponding to the product information image based on location information of a place where the product information image was photographed, and store the store information as attribute information as well.

For example, the product information retrieval apparatus 100 may vectorize and store product attribute information using a deep learning-based prediction algorithm.

Specifically, the product information retrieval apparatus 100 may vectorize ingredient and attribute information for each cosmetic type by using Word2Vec, and vectorize the distance between cosmetic ingredients by using a Continuous Bag-of-Words (CBOW) model. Since Word2Vec and CBOW are known technologies, detailed descriptions are omitted. This is to allow a product including a component that is close to the property information calculated based on the user's photographing record and search history to be recommended later.

The product information search apparatus 100 may capture a purchase receipt image or obtain it from an external device. Similarly in this case, the product information retrieval apparatus 100 may be a user terminal installed with an application that takes a purchase receipt image and performs a series of processes, or receives a purchase receipt image captured from the user terminal and performs a series of processes. However, for convenience of description, an example in which the product information retrieval apparatus 100 is implemented as a user terminal and obtains a purchase receipt image will be described as an example.

The purchase receipt contains information related to the purchase of various products. The purchase receipt photographed by the product information search device 100 may include various purchase-related information such as store information, product name, price, promotion information, quantity information, purchase time information, buyer information, and a unique identification code. .

5 is an exemplary diagram of a purchase receipt image.

The purchase receipt image refers to an image in which information capable of identifying a consumer's purchase details is photographed. For example, the purchase receipt image may include purchase information. Purchase information may include information on various objects such as store information, buyer information (name, phone number, etc.), product name, price, promotion information, quantity information, purchase time information, buyer information, and a unique identification code. The purchase receipt image may be an image of a paper receipt directly photographed offline or an image of an online receipt issued online.

The product information search apparatus 100 may identify purchase information from the captured purchase receipt image.

Specifically, the product information search apparatus 100 may specify at least one object in the purchase receipt image.

For example, referring to FIG. 5 , the product information retrieval apparatus 100 displays store information (Ob5), product name (Ob6), price (Ob7), promotion information (Ob8), quantity information (Ob9), and purchase information in a purchase receipt image. At least one of the time information Ob10 and the unique identification code Ob11 may be specified. At this time, the product information retrieval device 100 may specify an object using a convolutional neural network (CNN) model. As an example, the product information retrieval device 100 may use MobileNet ("MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H, arXiv:1704.04861, 2017.; MobileNetV2: Inverted Residuals and Linear Bottlenecks, Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC. arXiv preprint. arXiv:1801.04381, 2018.") An object can be specified by object detection technology based on the model.

The MobileNet model is a model based on a simplified structure using a convolutional layer divided in the depth direction. The convolutional product for the entire image is calculated, but factored into convolutional products per channel and 1 x 1 convolutional product, so that the purchase receipt image Included objects can be detected. In the case of using the MobileNet model, it is possible to detect objects at high speed while being relatively less constrained in terms of size and shape of images than conventional methods. In particular, the usability of the model in a user terminal such as a mobile device is increased.

Also, the product information search apparatus 100 may perform pre-processing on the purchase receipt image before specifying the object in order to increase the accuracy of the purchase receipt image. Pre-processing may be, for example, performing image binarization conversion, noise removal, surrounding background removal, resolution change, and the like.

Also, the product information search apparatus 100 may identify text in the captured purchase receipt image and extract text data. At this time, the product information search apparatus 100 may identify text by reading the entire image area or may identify text for each specified object.

As an embodiment, the product information search apparatus 100 may detect text in the purchase receipt image using an optical character recognition (OCR) method or the like.

The product information search apparatus 100 may perform post-processing on text data.

As an embodiment, the product information search apparatus 100 may post-process text data based on at least one of character unit information and text context information of the purchase receipt image.

For example, the product information retrieval apparatus 100 may infer, as promotion information, that a price discount event has occurred through a % (percent) or - (minus) sign in the purchase receipt image in which the promotion information is damaged.

In addition, the product information retrieval apparatus 100 may match information corresponding to a specific object by utilizing contextual information between surrounding texts in a purchase receipt image in which specific object information is damaged.

Also, the product information search apparatus 100 may specify a product of interest of the user who has acquired the purchase receipt image, specify service information corresponding to the product of interest, and provide the specified service information to the user.

Specifically, when the user captures a purchase receipt image of a specific product, the product information search apparatus 100 provides the user with a vector component identical to or similar to that of the specific product (eg, a vector value within a preset reference range). ) Other products may be provided as service information.

In addition, the product information search device 100 may provide store information where other products are sold and service information (eg, sales information of other cheaper stores or information about alternative products having the same/similar properties, etc.) .

In addition, the product information retrieval apparatus 100 acquires a purchase receipt image of a specific product in a state in which a plurality of product information images are obtained from other users and attribute information corresponding to each of the plurality of product information images is stored in a database. In one case, a product of interest of the user acquired from the purchase receipt image among various products accumulated on the database may be specified, and service information corresponding to the specified product of interest may also be specified.

6 is a diagram illustrating various examples of service information provided by a product search device.

For example, referring to (a) of FIG. 6 , the product information search device 100 stores attribute information on product A, product B, and product C as a user's offline log record in a database, and the user When the purchase receipt image of product A is acquired, it is provided as upper service information of other products that have similar property information to the user's product A, and comprehensive attribute information by comprehensively considering the property information of product A, product B, and product C. may be specified, and various other products having attribute information similar to the specified comprehensive attribute information may be provided as lower service information.

For another example, referring to (b) of FIG. 6 , the product information search device 100 stores attribute information on product B and product C as the user's offline log record in the database, and the user selects product A. When the purchase receipt image of is acquired, product B having attribute information most similar to product A of the user among product information stored in the database may be specified as service information.

Also, the product information retrieval apparatus 100 may associate the user's purchase information obtained through the purchase receipt image with attribute information on the user's interested product obtained through the product information image.

For example, referring to (c) of FIG. 6 , when the product information search device 100 stores the product name and promotion information for product A as attribute information through the product information image, the product information search device 100 uses the user's purchase receipt image User information (for example, age, gender, etc.) that matches the acquired purchase information (for example, personal information such as name or phone number) of the user is received from a separate personal information server, and the user information is stored in Product A It may be stored separately by matching with attribute information of . In this case, the product information search apparatus 100 may provide recommended service information for product A to other users (eg, 19-year-old male) having similar user information.

As described above, the product information search device 100 secures a large amount of learning data with product information images and purchase receipt images, and provides personalized service information to users, which is helpful in making purchase decisions. As the amount of information increases, the quality of service may increase.

The product information retrieval device 100 may include hardware resources and/or software required to implement the technical idea of the present invention, and does not necessarily mean one physical component or one device. no. That is, the photographing-based information providing system 100 may mean a logical combination of hardware and/or software provided to implement the technical idea of the present invention, and if necessary, it is installed in devices spaced apart from each other to provide It may be implemented as a set of logical components for implementing the technical idea of the present invention by performing functions.

In addition, the product information retrieval device 100 uses a memory for storing an algorithm for controlling the operation of components in the product information retrieval device 100 or a database for a program or application that reproduces the algorithm, and the database stored in the memory. It may include a processor that performs the above-described operation. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

Hereinafter, referring to FIG. 7 , a control method of the product information search device 100 will be described. 7 is a flowchart of a control method of a product information search apparatus according to an embodiment.

First, the product information retrieval apparatus 100 according to an embodiment obtains a purchase receipt image captured by the user terminal (1100).

Next, the product information retrieval apparatus 100 according to an embodiment identifies purchase information from the obtained purchase receipt image by using an artificial neural network model (1200).

Here, the operation 1200 of identifying purchase information includes an operation of specifying at least one object included in the purchase receipt image using a convolutional neural network (CNN) model, and extracting text data of the specified object. action may be included.

Purchase information may include information on various objects such as store information, buyer information (name, phone number, etc.), product name, price, promotion information, quantity information, purchase time information, buyer information, and a unique identification code.

Since the operation of specifying an object in the purchase receipt image has been described above in relation to MobilNet, redundant description will be omitted.

Since the operation of extracting text data has been described above in relation to the optical character reading method, redundant description will be omitted.

Also, the operation 1200 of identifying purchase information may further include an operation of pre-processing the purchase receipt image before specifying at least one object. The operation of preprocessing the purchase receipt image may include at least one of image binarization conversion, noise removal, surrounding background removal, and resolution change of the purchase receipt image.

The operation 1200 of identifying purchase information may further include an operation of post-processing the extracted text data. The post-processing operation may include post-processing the text data based on at least one of character unit information and text context information of the purchase receipt image. A detailed description of a method of post-processing text data based on at least one of character unit information and text context information will be omitted as described above.

Next, the apparatus 100 for retrieving product information according to an embodiment may specify a product of interest of a user whose purchase receipt image is captured based on the identified purchase information (1300).

The operation 1300 of specifying a product of interest may include an operation of specifying a product of interest of a user whose purchase receipt image is captured based on the extracted text data.

Also, the product information retrieval device 100 acquires through the user's purchase receipt image. Receive user information (eg, age, gender, etc.) that matches the user's purchase information (eg, buyer information such as the buyer's name or phone number) from a separate personal information server, and send the user information to the purchase receipt The image may be matched with the photographed product and stored in the database. In order to specify the product on which the purchase receipt image is photographed, the product information search device 100 may specify, as a product of interest, at least one product stored in the database having attribute information most similar to that of the product on which the purchase receipt image is photographed. there is. Attribute information of products stored in the database may be obtained in advance through product information images.

The operation 1300 of specifying the user's product of interest may include an operation of specifying the user's product of interest using a deep learning-based prediction algorithm. At this time, the deep learning-based prediction algorithm may include Word2Vec and Continuous Bag-of-Words (CBOW) models, and redundant descriptions thereof will be omitted.

Next, the product information search apparatus 100 may specify service information corresponding to the product of interest of the specified user (1400). In addition, the product information search apparatus 100 may provide specified service information to the user (1500).

The operation 1400 of specifying service information may include an operation of specifying, as service information, information on a recommended product having attribute information corresponding to attribute information of a product of interest.

For example, when the user captures a purchase receipt image of a specific product, the operation 1400 of specifying service information provides the user with a vector component identical to or similar to that of the specific product (eg, a vector within a preset reference range). values) may be provided as service information.

In this case, the product information retrieval apparatus 100 recommends a product preferred by another user (for example, a 19-year-old male) having similar user information to the user based on the user's purchase information included in the purchase receipt image. You can also provide information.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. Instructions may be stored in the form of program codes, and when executed by a processor, create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media in which instructions that can be decoded by a computer are stored. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, and the like.

As above, the disclosed embodiments have been described with reference to the accompanying drawings. Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a form different from the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

100: product information search device

Claims (13)

In the control method of a product information search device,
acquiring a purchase receipt image captured by a user terminal of a first user;
identifying purchase information from the purchase receipt image using an artificial neural network model;
specifying a product of interest of the first user who has captured the purchase receipt image based on the identified purchase information; and
specifying first service information for a recommended product having attribute information corresponding to the attribute information of the product of interest; and
providing the first service information to a user terminal of the first user;
Identifying the purchase information,
recognizing a plurality of objects included in the purchase receipt image using an artificial neural network model;
extracting text data of the plurality of objects; and
Post-processing text data extracted from the plurality of objects based on at least one of character unit information and text context information of the purchase receipt image;
In the post-processing step,
If the purchase receipt image includes a percent sign or subtraction code as the character unit information, based on the text context information and the percent sign or subtraction code, a price discount event has occurred for the product of the purchase receipt image Including the step of inferring as promotional information;
In the step of specifying the product of interest,
storing attribute information of a product on which the purchase receipt image is photographed in a database based on the extracted text data;
Receiving user information matched with purchase information of the first user obtained through the purchase receipt image, wherein the user information includes age and gender of the first user, from a separate personal information server;
Specifying, among a plurality of products stored in the database, at least one product having attribute information most similar to attribute information of a product on which the purchase receipt image is photographed as a product of interest to the first user; and
Matching the user information of the first user received from the personal information server with the product of interest and storing it in the database;
The control method of the product information search device,
providing second service information recommending a preferred product of a second user having user information of the same age and gender as the first user to a user terminal of the first user; and
providing third service information recommending a product of interest to the first user to the second user having user information of the same age and gender as the first user;
The control method of the product information search device further comprising. delete According to claim 1,
The artificial neural network model includes a convolutional neural network (CNN) model. According to claim 1,
The purchase information includes at least one of store information, product name, price, promotion information, quantity information, purchase time information, buyer information, and a unique identification code.
delete According to claim 1,
Prior to the step of identifying the purchase information,
The control method of the product information retrieval apparatus further comprising the step of performing pre-processing on the purchase receipt image. delete delete In the product information search device,
The product information search device includes a processor,
the processor,
Obtaining a purchase receipt image captured by a user terminal of a first user;
A plurality of objects included in the purchase receipt image are recognized using an artificial neural network model, text data of the plurality of recognized objects is extracted, and then based on at least one of character unit information and text context information of the purchase receipt image. By post-processing the text data extracted from the plurality of objects to identify purchase information in the purchase receipt image,
Specifying a product of interest of the first user who has captured the purchase receipt image based on the identified purchase information;
specifying first service information for a recommended product having attribute information corresponding to the attribute information of the product of interest;
Providing the first service information to the user terminal of the first user;
The post-processing of the extracted text data may include, when a percentage mark or subtraction code is included as the character unit information in the purchase receipt image, based on the text context information and the percent sign or subtraction code, the purchase receipt Including inferring as promotional information that a price discount event has occurred for the product of the image,
When the processor executes an operation of specifying the product of interest,
Based on the extracted text data, store attribute information of the product on which the purchase receipt image is photographed in a database;
Receiving user information matching purchase information of the first user acquired through the purchase receipt image, wherein the user information includes age and gender of the first user, from a separate personal information server;
Specifying, among the plurality of products stored in the database, at least one product having attribute information most similar to that of the product on which the purchase receipt image is photographed as a product of interest to the first user;
Matching the user information of the first user received from the personal information server with the product of interest and storing it in the database;
providing second service information recommending a preferred product of a second user having user information of the same age and gender as the first user to a user terminal of the first user;
and providing third service information recommending a product of interest to the first user to the second user having user information of the same age and gender as the first user. delete According to claim 9,
The artificial neural network model includes a convolutional neural network (CNN) model. A computer-readable recording medium on which a control method of the product information retrieval device according to any one of claims 1, 3, 4, and 6 is recorded. delete

Images