KR20180051448A - Method, apparatus and computer program for providing shopping informations - Google Patents

Abstract

The present invention relates to a method for allowing a service server to provide shopping information matched with an image displayed on a user device to a user device. The method comprises: step A of collecting product information to be sold in an online market and constituting a database for a product image describing a product; and step B of extracting a label from a query, retrieving the database based on the label, and determining a recommendation product similar to a product included in a page when receiving the query for the page displayed on the user device.

Description

[0001] METHOD, APPARATUS AND COMPUTER PROGRAM FOR PROVIDING SHOPPING INFORMATIONS [0002]

The present invention is directed to a method and apparatus for providing shopping information. More specifically, the present invention is directed to an advertising platform system that provides shopping information that matches an image displayed on a user device.

Due to the rapid development of computer processing technology, the Internet-based advertising industry has been developing to distribute user attributes to advertisements. The advertisement contents are not transmitted from the advertiser to the user in one direction, but the advertisements for the items of interest are provided by guessing the area of interest of the user.

For example, based on the purchasing history of the user, search history, etc., the user can guess the item of interest, and the advertisement can be provided based on the information. If the user has traditionally purchased corn and milk online, he or she may provide advertisements for corn, milk, or, if the user has searched for conventional sunglasses, provide the user with ads related to sunglasses.

However, there is a problem that these advertisements do not consider other contents at the time of exposure. Even if the user has already purchased corn and milk, if advertisements for corn and milk are displayed while browsing the contents of fashion magazine, the advertising effect for the user will be insignificant. As a result, advertisements irrelevant to the user's interest at that point in time can be provided.

On the other hand, displaying an advertisement having no relevance while browsing a specific company's web site may adversely affect the brand image of the corresponding company. A website operator can contract with an advertiser and earn profit by providing an advertisement area on his web site. In this case, there is concern that an advertisement for an item having no relation to his / her own content is loaded in the advertisement area.

For example, if a fashion magazine website displays ads about corn and milk along with fashion-related content, it can have a negative impact on the fashion magazine's brand image. Therefore, a company that considers the brand image as important does not provide the advertisement area of the website, and the advertisement market may be reduced.

It is an object of the present invention to provide an advertisement platform for providing shopping information matching with an image displayed on a user device.

It is another object of the present invention to provide a filtering method for searching matching products on an image basis and increasing the accuracy of image search result values.

A method for providing product information to a user device by a service server according to an embodiment of the present invention includes: an A step of collecting product information to be sold in an online market and constituting a database of a product image describing the product; And a step B, upon receiving a query for a page displayed on the user device, extracting a label from the query and searching the database based on the label to determine a recommended product similar to the product included in the page .

A method of providing product information to a user device according to another embodiment of the present invention includes collecting product information to be sold on an online market and analyzing the collected goods based on the similarity of the product image describing the product Extracting a label for describing the characteristics of the group by at least one group; Converting the label into a text form, tagging the converted text label to a product included in the group, and configuring a database; And receiving a query for a page displayed on the user device in a text form, searching for the text label of the database and determining a recommended product.

Further, a method of displaying merchandise information by a user device according to an embodiment of the present invention includes the steps of: displaying a search icon when an image included in a displayed page is searchable; A step B, upon receipt of a user selection input for the search icon, highlighting an area corresponding to an image object corresponding to the selected search icon, from another area; And a step C of layering and displaying a recommended commodity area for displaying recommended commodity information similar to the image object on the upper page of the displayed page.

According to the present invention, a product having a high relevance to a displayed image is recommended based on an image displayed on a user device, thereby significantly increasing the target exposure rate of the advertisement. In addition, according to the present invention, it is possible to filter objects that are not product recommendation targets in the process of extracting a label of an image, thereby enhancing the reliability of image search.

Figure 1 is a flowchart illustrating a platform system for providing advertisements in accordance with an embodiment of the present invention.
2 is a flowchart for explaining a process of building a database of product information and product description images according to an embodiment of the present invention
3 is a flowchart for explaining a process of determining a recommendation product based on a query image
4 is a view for showing a search icon on a displayed image and illustrating a user interface for advertising a recommended product

It is to be understood that the present invention is not limited to the description of the embodiments described below, and that various modifications may be made without departing from the technical scope of the present invention. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

Hereinafter, it is assumed that the user device displaying the product information is a mobile device, but the present invention is not limited thereto. That is, the electronic device in the present invention should be understood as a concept including all types of electronic devices capable of requesting a product search, such as a desktop, a smart phone, a tablet PC, and displaying search information.

It should also be noted that the concept of goods in this specification is not limited to the type of goods. That is, in this specification, a product should be understood as a concept including not only a type of object but also an intangible service that can be sold.

Further, the term " displayed page in an electronic device " in this specification includes a screen loaded on an electronic device and / or contents inside the screen so that the page can be immediately displayed on the screen according to a scroll of a user As shown in Fig. For example, in the display of the mobile device, the entire execution screen of the application extended in the horizontal or vertical direction and displayed according to the scrolling of the user can be included in the concept of the page, and the screen during the camera roll is also included in the concept of the page .

In the drawings, the same components are denoted by the same reference numerals. And in the accompanying drawings, some of the elements may be exaggerated, omitted or schematically illustrated. It is intended to clearly illustrate the gist of the present invention by omitting unnecessary explanations not related to the gist of the present invention.

1 is a flow diagram illustrating a platform system for providing advertisements in accordance with an embodiment of the present invention.

In step 110, the service server 10 collects product information, and in particular, builds a database of product images. According to the embodiment of the present invention, since an advertisement item is determined based on an image, the service server 10 constructs a database for a product image to be searched separately from other product information.

Meanwhile, the service server according to the embodiment of the present invention can construct a product image database by excluding an image that can serve as noise in image search for product recommendation.

For example, if a product is a hat and a hat and a woman are included in the product description image, a response to the hat product may be provided to the image query for the woman image, Can be created.

A more detailed description of a database for product information and product images in a service server according to an embodiment of the present invention and for filtering product images will be described later with reference to FIG. 2 attached hereto.

In step 210 of FIG. 2, the service server may collect product information.

For example, the service server includes a crawler, a parser, and an indexer to collect Web documents of an online store, and access text information such as a product image, a product name, and a price contained in the web document.

For example, a crawler may collect data from a Web site address in an online store, identify a Web site, and track links to deliver data related to product information to the service server. At this time, the parser analyzes the web document collected during the crawling process, extracts the product information such as the product image, commodity price, and product name included in the page, and the indexer can index the corresponding position and meaning.

Meanwhile, the service server can collect and index the product information from the website of the online store, but may also receive the product information from the server of the online store that is registered in advance.

In step 220, the service server may process the collected product images. This is for labeling the product based on the product image, and further classifying the product based on the similarity of the image, without relying on text information such as a product name or a sales category.

To do this, the service server extracts features of the product image and indexes the feature information of the images for efficiency.

More specifically, the service server can perform an Interest Point Detection of characteristic regions of product images. The feature region refers to a main region for extracting a descriptor, that is, a feature descriptor, of an image for judging whether or not there is a similarity among images.

According to the embodiment of the present invention, such a feature region is a region including a corner, a corner or the like out of the contour including the image, a blob distinguished from the surrounding region, a region which is unchanged or crooked according to the deformation of the image, It can be a dark or bright polar point and can be a patch (piece) of an image or an entire image.

Furthermore, the service server can extract the descriptor from the feature region (Descriptor Extraction). The feature descriptor is a vector value representing the characteristics of the image.

According to an embodiment of the present invention, the feature descriptor may be calculated using the position of the feature region for the image, or the brightness, color, sharpness, gradient, scale, or pattern information of the feature region. For example, the feature descriptor may be calculated by converting the brightness value, the brightness value, or the distribution value of the feature region into a vector.

According to the embodiment of the present invention, the feature descriptor for the image may include not only a local descriptor based on the feature region, but also a global descriptor, a frequency descriptor, a binary descriptor, It can be expressed as a Neural Network descriptor.

More specifically, the feature descriptor may include a global descriptor (e.g., a global descriptor) for converting the brightness, color, sharpness, gradient, scale, pattern information, etc. of each of the regions divided by the entire image or an arbitrary reference, Global descriptor).

For example, the feature descriptor may include a frequency descriptor for converting the number of times the specific descriptors classified into the image are included in the image, the number of inclusions of the global feature such as the previously defined color table, etc. into a vector value, A binary descriptor which extracts whether the size of each element value constituting the descriptor is larger or smaller than a specific value and then transforms it into an integer type and uses it in a layer of a neural network Or a neural network descriptor for extracting image information used for classification.

Particularly, according to the embodiment of the present invention, the feature information vector extracted from the product image can be converted into a lower dimension. For example, the feature information extracted through the artificial neural network corresponds to 4,000-dimensioned high-dimensional vector information, and it is appropriate to convert it into a low-dimensional vector in an appropriate range in consideration of resources required for search.

The feature information vector may be transformed using an algorithm such as PCA or the like, and the feature information converted into the low dimensional vector may be indexed into the corresponding product image.

Further, the service server can classify the products into at least one group by applying the machine learning technique based on the processed product description image, and extract meaning, i.e., label, for each product group. (Step 230)

According to the first embodiment of the present invention, the service server generates a neural network model that defines labels in advance and learns characteristics of images corresponding to the labels, classifies objects in the product images, and extracts labels of the objects can do. At this time, the service server can assign the corresponding label to the image matching the specific pattern and the random probability through the neural network model that has learned the pattern of the image corresponding to each label.

According to the second embodiment of the present invention, the service server learns the characteristics of the image corresponding to each label to form an initial neural network model, and applies a large number of product image objects thereto to further extend the neural network model have. Further, the service server may create a new group including the product if the product is not included in any group.

According to the first embodiment and the second embodiment, the service server can label the first group as a skirt, the third group as a dress, and the like as a second group under the feminine, and the style (short sleeves, long sleeves, ), The shape of the pattern, the material, the color, the abstract feeling (clean, chic, vintage, etc.), label the meaning of the item from the image object stochastically, .

According to the third embodiment of the present invention, the service server can form product groups in a hierarchical structure formed by a plurality of layers on the basis of whether or not the similarity of the product image features is similar without learning. Furthermore, a weight is assigned to the feature information of the product image according to the request of the layer, and the probability that the object is included in the specific product group can be calculated using the processed feature information.

In this case, further analysis may be required to associate the label that can be extracted from the group with a concept that can be recognized by a human being. For example, if the service server classifies products into three groups through image processing, extracts A for the first group of features, B for the second group of features, and C for the third group of features , It is necessary to interpret that A, B, and C mean, for example, feminine tops, blouses, and plaid patterns, respectively.

When using such a machine learning technique, the service server may collect feedback on the search results and update the distance and weight values of layers using feedback information. That is, the classification of product groups and / If feedback on the suitability of the product composition included in the group is provided, it can be used to update the parameters or the weights of the learning model.

Step 240 is a step of filtering the product explanation image. This is to constitute a product image database except the product image which can act as noise in the image search.

More specifically, the service server can determine whether to filter the product image by comparing the label extracted from the product image with the product category information obtained from the product information. When there are a plurality of images for a specific product, and the label extracted from one of the images is different from the sales category of the corresponding product, the corresponding image or a specific object in the image can be filtered.

For example, there is a first to third product image for the A product, and in the first product image (jacket for women), for the second product image (jacket for women) and (sunglasses, round) 3 In the product image (sunglasses, round), the label may be extracted. At this time, if the sales category of the product A is " sunglasses ", the service server can configure the product image database only by the second and third product images, excluding the first product image.

This filtering is intended to reduce image search noise. In the above example, if the A product is actually about the sunglasses and the database includes all of the first to third product description images, even if the query image is a jacket, it is determined that the A product for the sunglasses is similar to the first product image It is preferable to filter the first product image because the accuracy of the search may deteriorate.

In step 250, the service server can convert the extracted label into text and tag the corresponding product. This is to support text search on labels extracted from images.

For example, if the label extracted from the product description image is blue, plaid, or polarity, the service server tags the product with a label of blue, plaid, and polarity, and if the user searches for "plaid polarity" The product can be recommended. If such a function is used, a product that has not been searched can be provided as a recommended product if the seller of the goods does not attach the tag of the checkered pattern or the pollity to the product.

Returning to the description of FIG. 1, in step 120, the service server 10 may receive a template for specifying the advertisement area from the website server 20 that provides the advertisement area. If the web site server specifies an area where an advertisement is displayed on the web site through the template, the service server can design an advertisement SDK for displaying an advertisement for the advertisement area and deliver the advertisement SDK to the web site server. (Step 130) and the website server will then place the ad SDK on its web site.

The example shown in FIG. 1 is for an embodiment in which the advertising platform operates in the form of an SDK on a web site, but the present invention is not limited thereto. That is, the advertisement platform according to the embodiment of the present invention may be in the form of an application or plug-in installed in the mobile device, an ActiveX, an advertisement module, or the like, or may be software installed on the desktop or an extension program installed in the web browsing tool.

Then, when the user device accesses the website server and accesses predetermined or new content (step 140), the shopping SDK will be executed. (Step 150)

The shopping SDK can generate a query for inquiring shopping information and deliver it to the service server. (Step 160)

The query is for inquiring the service server about the shopping information, and may be a form of URL, image, text, etc.

For example, the shopping SDK of the user device may transmit a query including the history log of the web browser to the service server. The history log may include a browsing execution history of the web browser. In particular, the history log may transmit URL information of a web page executed at that time, that is, URL information of a web page provided by the web site server 20.

Further, the user device may extract the image, moving image, and text data included in the URL of the web page, and may transmit the extracted data to the query. Furthermore, if the URL, text, image, or video data can not be extracted, the screen shot can be extracted and sent to the query.

According to the preferred embodiment of the present invention, the user device can extract an object in the image requested by the user and transmit it to the query, and further can transmit the query inputted by the user to the query.

To this end, the user device may be operable to first determine whether a search for a displayed page is possible, and to display a search or result icon only when searchable. According to the above-described embodiment, when a product search is possible, the fact is informed to the user by using a search icon or the like, and the query is transmitted only when the user selects an icon, thereby reducing resources required for transmitting and processing a query.

More specifically, the user device uses the number of images included in the page displayed at the time point, the size of each of the images, the shape of each of the images, the text of the page, the possibility of image extraction and / It is possible to determine whether or not the contents included in the page can be searched and display the searched icon on the page only when it is possible.

A specific user interface for displaying the search icon will be described later with reference to FIG. 4 attached hereto.

On the other hand, in step 170, the service server can retrieve the product information based on the received query.

For example, when the service server receives text as a query, it can search for a product having a tag value that matches the text. According to an embodiment of the present invention, the tag may be a label extracted from a product image. Accordingly, even when a user requests a search based on a search word, i.e., a text, the search can be performed based on a product image rather than a description of a product arbitrarily described by the seller. That is, since the label extracted from the product description image is compared with the search word, the accuracy of the search is significantly improved.

According to a preferred embodiment of the present invention, the service server receives an image by query and compares the query image and the product image of the database to obtain a product whose feature information of the image is the same or similar, You can decide. To this end, a more detailed description of processing the query image by the service server will be described below with reference to FIG. 3 attached hereto.

3 is a diagram for explaining a process of determining a recommendation product based on a query image.

In step 310, the service server receives the query image, and if the query image includes a plurality of objects, each of the objects is divided and recognized. (Step 320)

Steps 330 to 350 are steps for filtering an image that is not a recommendation object from a plurality of objects included in the query image.

In step 330, the service server may filter based on the probabilistic reliability of the previously learned neural network model in the process of classifying the images included in the query image.

For example, if a service server only recommends a product to a fashion item, it may be efficient to process the query objects, but not the fashion items, such as cars, grass, buildings, roads, chairs, Furthermore, if there is no object related to the fashion item in the query image, the query image will not need to go through a search process for product recommendation.

Accordingly, the present invention proposes a method of filtering a query image based on inclusion and / or exclusion conditions.

More specifically, the service server can search the query image for a predetermined image pattern, that is, whether or not the inclusion condition pattern exists, and if not, filter the query image. (Step 340)

For example, in the above example of recommending a fashion item, a pattern of a shape of a human body such as a whole body, an upper half body, and a lower half body is specified in advance, and when a human body does not exist in a query image, It can be judged that it is not. Further, even when a human body exists in the query image, if the importance of the whole body of the query object is low considering the position, shape, and the like of the human body object, the query image can be determined as not a target object.

Further, the service server may search the query image for a predetermined image pattern, that is, whether the non-existent condition pattern exists, and may filter the query image if it does not exist. (Step 350)

For example, in the above example of recommending a fashion item, if a pattern for a form of objects other than a recommendation object such as a car, a lawn, a building, a road, a chair, etc. is specified in advance and the pattern is mainly included in a query image, Can be filtered.

Further, according to a further embodiment of the present invention, the importance of a corresponding object is calculated in consideration of a position or a shape of objects recognizable in the query image in the query image, The pattern may be filtered by a method of calculating the degree of similarity with the pattern of the non-existent condition.

Although steps 340 and 350 in FIG. 3 are shown as a continuous process, this is only an example. That is, according to the embodiment of the present invention, the steps 340 and 350 may be performed as separate independent processes and may be omitted.

In step 360, the service server may process the query image. This is for judging whether or not the image is the same as the product image database, and the service server can process the query image processing in substantially the same manner as in Fig. 2 described above.

For example, the service server extracts characteristic values of a query image and extracts corresponding product images belonging to the same group as the corresponding query image through an artificial neural network model generated by applying a machine learning technique to create a database have.

At this time, the service server may process feature values of the query image reflecting the weights requested by the product groups distributed in the plurality of layers in the database, and calculate at least one product group having a distance value within a certain range with the query image .

Also, at this time, the service server can set groups or labels corresponding to each other in advance even if the meaning of the query image and the products in the database are not directly related. For example, when a group or label of an object extracted from a query image is a man, a dull face, or a beard, the service server may recommend a preset beard management tool, a razor, and a skin care program to correspond to the object.

Returning to the description of FIG. 1, the service server searching for the recommended product in step 170 can determine an exposure product, that is, an advertisement product, among the recommended products in consideration of the priority of the product exposure in step 180. The priority may illustrate the similarity of the image features, the similarity of the group and / or the label, or whether it is the product of an advertiser affiliated with the service server.

The service server may then provide information about the advertisement product to the user device (step 190) and the user device may display the product in a pre-specified advertisement area.

A more specific user device for displaying an advertisement item on a user device is described below with reference to FIG. 4 attached hereto.

4 is a diagram illustrating a user interface for displaying a search icon on a displayed image and advertising a recommended product.

4A is displayed on the user device, the user device transmits the image to the service server as a query, determines 401, 402, 403, and 404 as the service server search object, .

In this case, the user device can display a search icon 410, such as 410, over the vicinity of 401, 402, 403 and 404. The search icon may provide a function of linking to a URL for a recommended product or displaying a list of search results.

On the other hand, if the user selects a detailed description icon 415 in the example of FIG. 4A, the search icon may be expanded as shown in FIG. 4B, and information cards for recommended products such as 405, 406, 407, and 408 may be displayed. The recommended product information card may include a label extracted by the service server from the object, a product name of the recommended product, a price of the recommended product, and a representative image of the recommended product.

According to another embodiment of the present invention, the recommended item information card may be provided only with a label for the corresponding object as shown in 421 to 426 shown in FIG. 4C.

In FIG. 4C, if the user selects the 426 search icon, the user device can switch to the screen as shown in FIG. 4D and display information about a product similar to the object.

According to another embodiment of the present invention, the recommended product information can be displayed by layering on the query image.

For example, if content such as 420 in Figure 4e is displayed on the user device, the user device may display a search icon, such as 430. [

If the user selects 430, as shown in FIG. 4F, the image processing unit 420 processes the brightness and darkness of the region different from the query image region 432 and processes it, and creates an upper layer 420 for displaying a recommended product such as 433.

More specifically, at least one or more search icons may be displayed while blinking until a user selection is received. If the user selects one of the plurality of search icons, the image object area corresponding to the selected icon is highlighted Can be displayed.

At this time, the highlighting may be diffused to the edge area of the object centered on the search icon, and may be displayed in the form of at least one rectangle including the edge of the object, or one or more layered icons obtained by cropping the object .

If the user selects the 433 layer and inputs the vertical drag gesture, the 433 layer may be extended or shortened according to the direction and speed of the user gesture. FIG. 4G illustrates a case where the 433 layers, that is, the recommended product layer, is completely encapsulated and displayed as 420 layers.

FIG. 4G illustrates a layer that can extend in the vertical direction, but the present invention is not limited to this. That is, the recommended product display layer may extend in the horizontal direction. In this case, if the user selects the 433 layer and inputs a drag gesture in the horizontal direction, the content included in the 433 layer can be displayed in a fixed form by moving in the horizontal direction according to the direction and speed of the user gesture.

Further, the recommended product area according to the embodiment of the present invention may be displayed by being layered on the original content as shown in FIG. 4F, but may be displayed integrally with the original content. That is, since the advertisement platform according to the embodiment of the present invention can designate an arbitrary region in the content template in advance as a recommended product region, it can display the recommended product information in the designated region.

Further, the recommended product area includes a category area 470 for displaying at least one label 472, 473, 474 that can be extracted from the selected image object as shown in FIG. 4H, and thumbnail images 482, 485 of the recommended products And a recommended goods image link area 480.

5 is an illustration of a product information display user interface according to another embodiment of the present invention.

According to an embodiment of the present invention, an image displayed on the user device is a search target, as shown in FIG. 5A, and the image does not matter what type of application is executed. That is, the image can be displayed through a gallery, a web browsing application as well as other applications.

FIG. 5B is an illustration of a user interface that an application according to an embodiment of the present invention invokes to the Notee panel to activate a screen shot creation and / or camera shooting function. The user can generate a search target image by selecting a screen shot generating icon and / or a camera shooting icon generated in the Notee panel as shown in FIG. 5B.

5C is an illustration of a user interface for specifying and displaying a searchable image object among displayed images. A service server according to an exemplary embodiment of the present invention can specify a searchable image object in a query received from a user device and provide related information to the user device. The user device highlights searchable image objects in the displayed image, can do. The user can select an object to be searched among the highlighted image objects as shown in FIG. 5C.

5D is an example of a user interface for providing recommended product information. The recommended product may be provided in the form of a thumbnail for the representative product image. When a user selects a specific thumbnail, a detailed product description page can be created.

The embodiments of the present invention disclosed in the present specification and drawings are intended to be illustrative only and not intended to limit the scope of the present invention. It will be apparent to those skilled in the art that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein.

Claims (4)

A method for a service server to provide product information to a user device,
A step of collecting product information to be sold in an online market, the product information including a product image describing the product, and a database for the product; And
The method comprising: extracting a label describing a feature of a query image included in the query from the query, searching the database based on the label, And a step B for determining a recommended product,
In the step A,
Collecting at least one product image and at least one category information for an arbitrary commodity; And
Extracting at least one label from an object extracted from the product image, comparing the label with the category information, and configuring a database for the product by filtering the product image when the label and the category information are different And providing the goods information. A method for a service server to provide product information to a user device,
A method for collecting product information to be sold in an online market, sorting the collected products based on the similarity of the product images describing the products into at least one group, and extracting a label for describing the characteristics of the group for each group step;
A step B for converting the label into a text form, tagging the converted text label on a product included in the group, and constructing a database; And
Receiving a query for a page displayed on the user device in a text form, and searching for the text label of the database to determine a recommended product,
In the step A,
Collecting at least one product image and at least one category information for an arbitrary commodity; And
Extracting at least one label from an object extracted from the product image, comparing the label with the category information, and configuring a database for the product by filtering the product image when the label and the category information are different And providing the goods information. A method of displaying an advertisement,
The service server setting a product recommendation target pattern of an image object preset as a product recommendation target;
The service server receiving a query including an image displayed on a user device;
B) the service server extracting at least one or more image objects included in the query, and transmitting recommendation item information including a product image similar to the image object to the user device; And
Wherein the user device displays the recommended goods information,
The step (b)
Calculating a degree of similarity between the image objects extracted from the query and the product recommendation target pattern; And
And determining that the image object is not an object of recommendation if the degree of similarity is less than a set range. The service server setting a product recommendation target pattern of an image object preset as a product recommendation target;
The service server receiving a query including an image displayed on a user device;
Wherein the service server extracts at least one image object included in the query and calculates a similarity degree between the extracted image objects and the product recommendation target pattern, and when the similarity is less than the set range, ;
Extracting a label for describing a characteristic of an image object of the query, searching for a product information database based on the label, and recommending product information including a product image similar to the image object, Transmitting to the user device; And
And the user device displays the recommended product information.

Images