KR20200058343A - Method, apparatus and computer program for providing shopping informations - Google Patents

Abstract

The present invention relates to a method for providing product information to a user device by a service server. The method capable of improving reliability of image search comprises: a step A of collecting information on a product sold in an online market and constructing a database for a product image describing the product; and a step B of extracting a label from a query and searching the database based on the label to determine a recommended product similar to the product included in a page when a query on the page displayed on a user device is received.

Description

How to provide shopping information, devices and computer programs {METHOD, APPARATUS AND COMPUTER PROGRAM FOR PROVIDING SHOPPING INFORMATIONS}

The present invention relates to a method and apparatus for providing shopping information. More specifically, the present invention relates to an advertising platform system that provides shopping information matching an image displayed on a user device.

With the rapid development of computer processing technology, the Internet-based advertising industry has recently developed to distribute advertisements by reflecting user attributes. The advertisement content is not delivered in one direction from the advertiser to the user, but the user's area of interest is guessed so that the advertisement for the item of interest is provided.

For example, based on the user's purchase history, search history, and the like, the user's interest item can be guessed, and an advertisement may be provided based on this. If the user has previously purchased corn and milk online, the advertisement is provided to the user for corn and milk, or if the user searches for conventional sunglasses, the user is provided with advertisements related to sunglasses.

However, such advertisements have a problem of not considering other content at the time of exposure. Even if the user has previously purchased corn and milk, if the advertisement for corn and milk is displayed while browsing the contents of the fashion magazine, the advertising effect for the user will be negligible. That is, as a result, advertisements irrelevant to the user's interests may be provided.

On the other hand, the display of advertisements that are not relevant at all during browsing a website of a specific company may negatively affect the brand image of the corresponding company. A website operator can earn money by signing a contract with an advertiser and providing an advertisement area on his website, and may be concerned that advertisements for items irrelevant to his content are mounted in the advertisement area.

For example, an advertisement for corn and milk along with fashion-related content on a fashion magazine website could adversely affect the brand image of the fashion magazine. Therefore, companies that consider the brand image as important do not provide the advertising area of the website, and the advertising market may shrink.

The present invention is to solve the above problems, and an object of the present invention is to provide an advertising platform that provides shopping information matching an image displayed on a user device.

Furthermore, an object of the present invention is to provide a filtering method for searching for a matching product based on an image and increasing the accuracy of the image search result value.

A method of providing product information to a user device by a service server according to an embodiment of the present invention comprises: A step of collecting product information sold in an online market and configuring a database for product images describing the product; And receiving a query for a page displayed on the user device, extracting a label from the query, and searching the database based on the label to determine a recommended product similar to the product included in the page. Is done.

A method of providing product information to a user device by a service server according to another embodiment of the present invention collects product information sold in an online market, and collects the product based on the similarity of the product image describing the product. Classifying into at least one group, and extracting a label for each group to describe characteristics of the group; Converting the label into a text form and tagging the converted text label into a product included in a corresponding group to construct a database; And when a query for a page displayed on the user device is received in text form, retrieving the text label of the database to determine a recommended product.

Furthermore, a method of displaying product information by a user device according to an embodiment of the present invention includes: A step of displaying a search icon when searching for an image included in a displayed page is possible; Receiving a user selection input for the search icon, displaying a region corresponding to the image object corresponding to the selected search icon by highlighting the region than other regions; And a step C of displaying the recommended product area for displaying recommended product information similar to the image object by layering on the displayed page.

According to the present invention, based on the image displayed on the user device, a product having a high relevance to the displayed image is recommended, and the target exposure rate of the advertisement is significantly increased. Furthermore, according to the present invention, objects that are not targeted for product recommendation can be filtered in the process of extracting the label of the image, thereby increasing the reliability of the image search.

1 is a flowchart illustrating a platform system for providing advertisements according to an embodiment of the present invention
Figure 2 is a flow chart for explaining the process of building a database for product information and product description image according to an embodiment of the present invention
3 is a flowchart illustrating a process of determining a recommended product based on a query image
4 is a diagram for illustrating a user interface displaying a search icon on a displayed image and advertising a recommended product

The present invention is not limited to the description of the embodiments described below, and it is apparent that various modifications can be made without departing from the technical spirit of the present invention. In describing the embodiments, descriptions of technical contents that are widely known in the technical field to which the present invention pertains and which are not directly related to the technical subject matter of the present invention will be omitted.

Hereinafter, it is assumed and described that the user device on which the product information is displayed is a mobile device, but the present invention is not limited thereto. That is, in the present invention, the electronic device should be understood as a concept including all types of electronic devices capable of requesting product search, such as a desktop, smartphone, or tablet PC, and displaying search information.

It should also be noted that the concept of goods in this specification is not limited to tangible goods. That is, in this specification, a product should be understood as a concept including tangible goods as well as intangible services that can be sold.

Further, in this specification, the term displayed on an electronic device (displayed page in an electronic device) includes a screen loaded on the electronic device and / or content inside the screen so that it can be immediately displayed on the screen according to the user's scroll. Can be understood as a concept. For example, in a display of a mobile device, an entire execution screen of an application that is extended in a horizontal or vertical direction and displayed according to a user's scroll may be included in the concept of the page, and the screen being rolled by the camera may also be included in the concept of the page. Can be.

Meanwhile, in the accompanying drawings, the same components are represented by the same reference numerals. In the accompanying drawings, some components may be exaggerated, omitted, or schematically illustrated. This is to clearly describe the gist of the present invention by omitting unnecessary descriptions not related to the gist of the present invention.

1 is a flowchart illustrating a platform system for providing advertisements according to an embodiment of the present invention.

In step 110, the service server 10 may collect product information and, in particular, build a database for product images. According to an embodiment of the present invention, since an advertisement item is determined based on an image, the service server 10 constructs a database for a product image to be searched separately from other product information.

Meanwhile, the service server according to an embodiment of the present invention may configure a product image database except for an image that may act as noise in image search for product recommendation.

For example, if the product for sale is a hat, and the product description image includes a hat and a women's top, a response to the hat product may be provided in the image query for the women's top, and the product image database is excluded. You can write

A more detailed description of building a database for product information and product images in a service server according to an embodiment of the present invention and filtering the product images will be described later with reference to the accompanying FIG. 2.

In step 210 of FIG. 2, the service server may collect product information.

For example, the service server may include a crawler, a parser, and an indexer, collect web documents of online stores, and access text information such as product images, product names, and prices included in web documents.

For example, a crawler may collect a list of web addresses of online stores, check a website, and track links to deliver data related to product information to a service server. At this time, the parser analyzes the web document collected during the crawling process and extracts product information such as product images, product prices, and product names included in the page, and the indexer can index the corresponding location and meaning.

Meanwhile, the service server may collect and index product information from the website of the online store, but may also receive product information from the server of the online store registered in advance.

In step 220, the service server may process the collected product image. This is for not only relying on text information such as a product name or a sales category, but also for labeling the product based on the product image and further classifying the product based on the similarity of the image.

To this end, the service server can extract the feature of the product image and index the feature information of the images for search efficiency.

More specifically, the service server may detect feature areas of product images (Interest Point Detection). The feature region refers to a main region for extracting a feature descriptor, i.e., a feature descriptor, for an image feature to determine whether the images are identical or similar.

According to an embodiment of the present invention, such a feature region may be a contour that an image includes, a corner such as a corner among contours, a blob separated from the surrounding region, an area that is invariant or covariant according to image deformation, or ambient brightness. It can be a pole with dark or light features, and it can be a patch (fragment) of the image or the entire image.

Furthermore, the service server can extract feature descriptors from the feature area. The feature descriptor expresses the features of the image as vector values.

According to an embodiment of the present invention, such a feature descriptor can be calculated using the location of the feature region for the corresponding image, or the brightness, color, sharpness, gradient, scale, or pattern information of the feature region. For example, the feature descriptor may calculate the brightness value of the feature region, the change value of the brightness, or the distribution value by converting it into a vector.

On the other hand, according to an embodiment of the present invention, the feature descriptor for the image is a local descriptor based on the feature area as described above, as well as a global descriptor, a frequency descriptor, a frequency descriptor, a binary descriptor, or the like. It can be expressed as a neural network descriptor.

More specifically, the feature descriptor is a global descriptor that converts and extracts the brightness, color, sharpness, gradient, scale, pattern information, etc. of each image or each region of the image divided by arbitrary criteria into vector values ( Global descriptor).

For example, the feature descriptor is a frequency descriptor (Frequency Descriptor) that converts and extracts the number of specific descriptors previously included in the image and the number of global features such as a previously defined color table into a vector value and extracts them. Binary descriptor used to extract whether it is included or if the size of each element value constituting a descriptor is greater than or less than a specific value in bits, and then convert it to an integer to learn in the layer of the neural network Or it may include a neural network descriptor (Neural Network descriptor) for extracting the image information used for classification.

In particular, according to an embodiment of the present invention, it is possible to transform the feature information vector extracted from the product image into a lower dimension. For example, the feature information extracted through the artificial neural network corresponds to 40,000-dimensional high-dimensional vector information, and it is appropriate to convert to a low-dimensional vector in an appropriate range in consideration of resources required for search.

An algorithm such as PCA may be used to convert the feature information vector, and feature information converted into a low dimensional vector may be indexed into a corresponding product image.

Furthermore, the service server may classify products into at least one group by applying a machine learning technique based on the processed product description image, and extract meanings, that is, labels, for each product group. (Step 230)

According to the first embodiment of the present invention, the service server defines a label in advance, generates a neural network model learning the characteristics of the image corresponding to the label, classifies objects in the product image, and extracts the label of the object can do. At this time, the service server may assign a corresponding label to an image that matches a specific pattern with a random probability through a neural network model learning the pattern of the image corresponding to each label.

According to the second embodiment of the present invention, the service server may learn characteristics of an image corresponding to each label to form an initial neural network model, and apply a large amount of product image objects to expand the neural network model more precisely. have. Furthermore, the service server may create a new group including the product if the product is not included in any group.

According to the first and second embodiments, the service server may label the first group as women, the second group as skirts, and the third group as dresses, and as subgroups of styles (short sleeves, long sleeves, etc.) ), The shape of the pattern, material, color, and abstract feeling (innocent, chic, vintage, etc.) can be additionally labeled, and the meaning of the product can be calculated from the image object stochastically, and it can be labeled on the product in the group. .

According to the third embodiment of the present invention, the service server may form product groups in a hierarchical structure formed of a plurality of layers based on whether the product image features are the same or similar without separate learning. Furthermore, weights can be assigned to the feature information of the product image according to the request of the corresponding layer, and the probability that the corresponding object is included in a specific product group can be calculated using the processed feature information.

In this case, additional analysis may be necessary to link the label that can be extracted from the group with a concept that can be recognized by a real human. For example, when the service server classifies products into three groups through image processing, and extracts labels A for characteristics of the first group, B for characteristics of the second group, and C for characteristics of the third group. It is necessary to interpret that A, B, and C mean, for example, female tops, blouses, and checkers, respectively.

When using such a machine learning technique, the service server collects feedback on the search results and can update the distance and weight values of the layers using the feedback information, that is, classification of product groups and / or corresponding products If feedback is provided on the suitability of product composition included in the group, the learning model may be updated with parameters or weights.

Step 240 is a step of filtering the product description image. This is to construct a product image database excluding product images that may act as noise in image search.

More specifically, the service server may determine whether to filter the product image by comparing the label extracted from the product image and the product category information obtained from the product information. If there are multiple images for a specific product, and the label extracted from one of the images differs from the sales category of the product, the image or a specific object within the image may be filtered.

For example, there are first to third product images for product A, in the first product image (women's tops, jackets), in the second product image (women's tops, jackets) and (sunglasses, rounds), products 3 In the product image, consider the case where the label (Sunglasses, Round) is extracted. At this time, if the sales category of the A product is “sunglasses”, the service server may configure the product image database with only the second and third product images, excluding the first product image.

This filtering is intended to reduce noise in image search. In the above example, if product A is actually about sunglasses, and the database is configured to include all of the first to third product description images, it is determined that the query image is similar to the first product image even if it is a jacket, so product A for sunglasses is It is recommended to filter the first product image because the accuracy of the search may be reduced.

In step 250, the service server may convert the extracted label into text and tag the corresponding product. This is to support text search for labels extracted based on images.

For example, if the label extracted from the product description image is blue, checkered, poloty, the service server tags the product with the label blue, checkered, poloty, and when the user searches for “checkered poloti”, the tag is searched first. You can recommend the product. Using such a function, when a product seller does not have a tag for a checkered pattern or a polo on the product, there is an effect that a product that has not been searched can be provided as a recommended product.

Returning to the description of FIG. 1 again, in step 120, the service server 10 may receive a template for specifying the advertisement area from the website server 20 providing the advertisement area. When the website server specifies an area in which the advertisement is displayed on the website through the template, the service server may design an advertisement SDK for displaying advertisements for the advertisement area and deliver it to the website server. After step 130, the website server will mount the advertisement SDK on its website.

The example shown in FIG. 1 is for an embodiment in which the advertisement platform operates in the form of an SDK on a website, but the present invention is not limited thereto. That is, the advertisement platform according to an embodiment of the present invention may be in the form of an application or plug-in installed on a mobile device, ActiveX, an advertisement module, or an extension program installed on a software installed on a desktop or a web browsing tool.

Then, when the user device accesses the website server to access the specified or new content in advance (step 140), the shopping SDK will be executed. (Step 150)

The shopping SDK may generate a query for inquiring shopping information and deliver it to a service server. (Step 160)

The query is for inquiring the shopping information to the service server, and may exemplify the form of URL, image, text, etc.

For example, the shopping SDK of the user device may send a query including the history log of the web browser to the service server. The record log may include a browsing history of a web browser, and in particular, URL information of a web page executed at a corresponding time, that is, URL information of a web page provided by the website server 20 may be transmitted as a query.

Furthermore, the user device may extract image, video, and text data included in the URL of the web page, and transmit the extracted data as a query. Furthermore, if URL, text, image, or video data cannot be extracted, a screenshot can be extracted and sent as a query.

According to a preferred embodiment of the present invention, the user device may extract an object in an image requested by the user for search, and transmit the query as a query.

To this end, the user device may first determine whether or not a search for a displayed page is possible, and operate to display a search or result icon only when it is searchable. According to the above embodiment, when product search is possible, the fact that the user is notified using a search icon or the like and the query is transmitted only when the user selects the icon has an effect of reducing the resources required to send and process the query.

More specifically, the user device is included in the page displayed at the time, using the number of images, the size of each of the images, the shape of each of the images, the text of the page, whether the image can be extracted, and / or the form, address, etc. of the URL. It is possible to determine whether or not a search for content included in the page is possible, and a search icon can be displayed on the corresponding page only when possible.

The detailed user interface for displaying the search icon will be described later with description of the attached FIG. 4.

Meanwhile, in step 170, the service server may search for product information based on the received query.

For example, when receiving text in a query, the service server may search for a product having a tag value matching the text. According to an embodiment of the present invention, the tag may be a label extracted from a product image. Accordingly, even when a user requests a search based on a search word, that is, text, it may be searched based on a product image rather than a product description arbitrarily written by the seller. That is, since the label extracted from the product description image and the search term are compared, the accuracy of the search is greatly improved.

According to a preferred embodiment of the present invention, the service server receives an image through a query and compares the query image with the product image in the database, so that the product having the same or similar characteristic information of the image or the configuration of the label is a recommended product. Can decide. To this end, a more detailed description of processing a query image by the service server will be described later with reference to the attached FIG. 3.

3 is a diagram for explaining a process of determining a recommended product based on a query image.

In step 310, the service server may receive a query image, and if a plurality of objects are included in the query image, each of the objects may be divided and recognized. (Step 320)

Steps 330 to 350 are steps for filtering an image that is not a recommendation target from a plurality of objects included in the corresponding query image.

In step 330, the service server may filter based on the stochastic reliability of the neural network model previously learned in the process of classifying the image included in the query image.

For example, when a service server recommends a product limited to a fashion item, it may be efficient to process query objects except image objects that are not fashion items, for example, cars, lawns, buildings, roads, and chairs. Furthermore, if there is no object related to the fashion item in the query image, the query image will not need to go through a search process for product recommendation.

Therefore, the present invention proposes a method of filtering a query image based on an inclusion condition and / or an inclusion condition.

More specifically, the service server may search for a pre-specified image pattern in the query image, that is, whether an inclusion condition pattern exists, and if not, filter the corresponding query image. (Step 340)

For example, in the previous example of recommending a fashion item, the pattern for the shape of the human body, such as the whole body, upper body, and lower body, is specified in advance, and if the human body does not exist in the query image, the query image is recommended. It can be judged not to be. Furthermore, even if a human body exists in the query image, if the importance is low considering the position and shape of the human body object in the entire query image, it may be determined that the query image is not recommended.

Furthermore, the service server may search for a pre-specified image pattern in the query image, that is, whether or not an inclusion condition pattern exists, and filter the corresponding query image if it does not exist. (Step 350)

For example, in the previous example of recommending a fashion item, if a pattern for shapes of objects that are not recommended, such as a car, a lawn, a building, a road, or a chair is specified in advance, and the pattern is mainly included in the query image, the corresponding query image Can be filtered.

Further, according to a further embodiment of the present invention, the importance of the corresponding object is calculated in consideration of the position or shape of objects in the query image that are recognizable in the query image, and only the objects having the importance of a specific range or more are included You can also filter by calculating the similarity between the pattern and the condition not included.

Although steps 340 and 350 are shown in FIG. 3 as a continuous process, this is only an example. That is, according to an embodiment of the present invention, steps 340 and 350 may be performed as separate independent processes or may be omitted.

In step 360, the service server may process the query image. This is for determining whether the images are the same as compared with the product image database, and the service server can perform query image processing in substantially the same manner as in FIG. 2 described above.

For example, the service server can extract product images belonging to or corresponding to the same group as the corresponding query image through the artificial neural network model generated by applying the machine learning technique to extract the feature values of the query image and create a database. have.

At this time, the service server may process the feature values of the query image by reflecting the weights required by the product groups distributed in a plurality of layers in the database, and calculate at least one product group having a distance value within a certain range from the query image. .

In addition, at this time, the service server may set a group or label corresponding to each other in advance even if the meaning of the query image and the product in the database are not directly related. For example, if the group or label of objects extracted from the query image is a man, a dull face, and a beard, the service server may recommend a preset beard management tool, a razor, and a skin care program to correspond to the object.

Returning to the description of FIG. 1 again, the service server searching for the recommended product in step 170 may determine the exposed product, that is, the advertisement product among the recommended products in consideration of the priority of product exposure in step 180. The priority may exemplify similarity of image characteristics, similarity of groups and / or labels, or whether the product is an advertiser affiliated with the service server.

Thereafter, the service server may provide information about the advertisement product to the user device (step 190), and the user device may display the corresponding product in a predetermined advertisement area.

A more specific user device displaying the advertisement product on the user device will be described later with the description of FIG. 4 attached.

FIG. 4 is a diagram illustrating a user interface displaying a search icon on a displayed image and advertising a recommended product.

When the image shown in FIG. 4A is displayed on the user device, the user device transmits the image to the service server by query, determines 401, 402, 403, 404 as the object to be searched for the service server, and recommends products similar to the object. You can decide.

In this case, the user device may display the search icons, such as 410, by overlaying them near 401, 402, 403, and 404. The search icon may provide a function of linking to a URL for a recommended product or displaying a list of searched results.

On the other hand, if the user selects a detailed description icon such as 415 in the example of FIG. 4A, the search icon is expanded as shown in FIG. 4B, and an information card for recommended products such as 405, 406, 407, and 408 may be displayed. The recommended product information card may include a label extracted by the service server from the object, a product name of the recommended product, a price of the recommended product, and a representative image of the recommended product.

Meanwhile, according to another embodiment of the present invention, the recommended product information card may be provided only with a label for the corresponding object, such as 421 to 426 shown in FIG. 4C.

When the user selects the 426 search icon in FIG. 4C, the user device may switch to a screen as shown in FIG. 4D and display information about a product similar to the corresponding object.

According to another embodiment of the present invention, recommended product information may be displayed by layering on a query image.

For example, in FIG. 4E, when content such as 420 is displayed on the user device, the user device may display a search icon such as 430.

When the user selects 430, as shown in FIG. 4F, 420 may be processed by classifying the contrast between the query image area 432 and other areas, and an upper layer for displaying recommended products such as 433 may be generated over 420.

More specifically, the search icon may be displayed at least one while blinking until receiving a user selection. When a user selects any one of a plurality of search icons, the image object area corresponding to the selected icon is highlighted than the other areas. Can be displayed.

At this time, the highlighting may be displayed by spreading to the edge region of the object centered on the search icon, and displayed in a minimum square shape including the edge of the object or in the form of one or more layered icons that crop the object. It may be.

At this time, if the user selects the 433 layer and inputs a drag gesture in the vertical direction, the 433 layer may be extended or shortened according to the direction and speed of the user gesture. FIG. 4G illustrates a case in which the 433 layer, that is, the recommended product layer is completely encroached on the 420 layer.

4G illustrates a layer that can be extended in the vertical direction, but the present invention is not limited thereto. That is, the recommended product display layer may be extended in the horizontal direction. In this case, when the user selects the 433 layer and inputs a drag gesture in a horizontal direction, the 433 layer is fixed in accordance with the direction and speed of the user gesture, and the content included in the 433 layer can be displayed by moving in the horizontal direction.

Further, the recommended product area according to the embodiment of the present invention may be displayed by being layered on the original content as shown in FIG. 4F, but may be displayed integrally with the original content. That is, since the advertisement platform according to the embodiment of the present invention can designate an arbitrary area in advance in the corresponding content template and designate it as a recommended product area, it is also possible to display recommended product information in the designated area.

Furthermore, the recommended product area is a category area 470 that displays at least one label 472, 473, 474 that can be extracted from the selected image object, as shown in FIG. 4H, and displays thumbnail images 482, 485 of recommended products. It may be configured to include the recommended product image link area 480.

5 is an illustration of a product information display user interface according to another embodiment of the present invention.

According to an embodiment of the present invention, as shown in FIG. 5A, an image displayed on a user device becomes a search target, and the image is irrespective of the type of an application that executes it. That is, the image may be displayed through a gallery, a web browsing application, or other applications.

5B is an example of a user interface in which an application according to an embodiment of the present invention calls a notification panel to activate a screenshot and / or camera shooting function. As illustrated in FIG. 5B, the user may select a screenshot creation icon and / or a camera shooting icon generated in the notty panel to generate an image to be searched.

5C is an example of a user interface for specifying and displaying a searchable image object among displayed images. The service server according to an embodiment of the present invention may specify a searchable image object in a query received from the user device, provide relevant information to the user device, and the user device highlights and displays searchable image objects in the displayed image can do. As illustrated in FIG. 5C, the user may select an object to request a search from among the highlighted image objects.

5D is an example of a user interface that provides recommended product information. The recommended product may be provided in the form of a thumbnail of the representative product image. When a user selects a specific thumbnail, a detailed product description page may be generated.

The embodiments of the present invention disclosed in the present specification and drawings are merely to provide a specific example to easily explain the technical content of the present invention and to understand the present invention, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art to which the present invention pertains that other modified examples based on the technical idea of the present invention can be implemented in addition to the embodiments disclosed herein.

Claims (4)

A method for displaying product information on a user device,
Upon receiving a user input requesting a search for an image included in the displayed page, at least one image object corresponding to a search target among the images is determined, and the search icon for the image object is overlaid on the determined image object. Displaying;
When a user selection input for the search icon is received, highlighting is displayed by spreading to the edge area of the image object centered on the search icon, and corresponds to the image object in a minimum square shape including the edge of the image object Displaying an area to be distinguished from other areas; And
And displaying a recommended product area for displaying recommended product information similar to the image object,
The step of displaying the division,
Calculating the importance of the first image object in consideration of the location and shape of the plurality of first image objects included in the displayed page;
A method of displaying product information, wherein among the second image objects, the importance of which is greater than or equal to a predetermined range among the first image objects, an image object different from a preset representative pattern is filtered to specify the search target. According to claim 1,
The recommended product area includes at least one of a category area displaying at least one label that can be extracted from the image object and a recommended product image link area displaying thumbnail images of the recommended products. How to display information. The method of claim 2, wherein the displaying of the recommended product area comprises:
A method of displaying product information, characterized in that a layer displaying the recommended product area is extended or reduced in a vertical or horizontal direction according to the direction and speed of a user gesture for the recommended product area. In the method for the user device to display the product information,
Upon receiving a user input requesting a search for an image included in the screen of the user device, at least one image object corresponding to a search target among the images is determined, and the image object is overlaid on the determined image object to search for the image object Displaying an icon;
When a user selection input for the search icon is received, highlighting is displayed by spreading to the edge area of the image object centered on the search icon, and corresponds to the image object in a minimum square shape including the edge of the image object Displaying an area to be distinguished from other areas;
Displaying a recommended product area for displaying recommended product information similar to the image object; And
And when a gesture in the left-right direction or the up-down direction is received in the recommended product area, displaying a list of recommended products corresponding to the gesture direction.
In the step of displaying, the importance of the first image object is calculated in consideration of positions and shapes of the plurality of first image objects included in the image requesting the search, and among the first image objects And filtering the image object different from the representative pattern preset as a product recommendation target among the second image objects whose importance is greater than or equal to a preset range,
The recommended product area includes one or more thumbnail images selectable by a user, wherein the product information is displayed.

Images