KR101715036B1 - Method for searching product classification and providing shopping data based on object recognition, server and system thereof - Google Patents

Abstract

The present invention relates to a shopping information providing service in an online shopping mall. In the shopping mall, when a user takes a picture of a shopping mall, And identifies the type of the interested goods and the interested goods contained in the image, extracts the shopping information about the interested goods, and provides the shopping purchase service through object recognition by providing the shopping information to the corresponding terminal.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a service providing method, a service providing method,

The present invention relates to a shopping information providing service method and system based on object recognition, and more particularly, to a shopping information providing method and system based on object recognition, in which, when searching for a desired item of interest in a predetermined terminal or a web site, A server and a system for providing goods classification search and shopping information through object recognition, which can additionally provide shop information of the same or similar goods as the image of the object of interest within the corresponding type will be.

Recent developments in communication technology and the activation of the Internet have led to various searches.

For example, an online shopping mall using the Internet accesses a specific web site, searches for a desired product, and purchases the product through price comparison. Recently, in the Internet environment, smart phones have been used to search for and purchase desired products using mobile terminals.

In addition, in the retrieval method, image based retrieval is proposed in a retrieval centered on a text such as an existing keyword. The image-based search has an advantage of searching for a desired product if the user does not know the exact keyword of the desired product but only the image of the product.

However, as the number of kinds of products has increased and the number of products has also increased, the work of image-based search or object search has become complicated and not done quickly. For example, if an image search is performed based on a database storing tens of thousands of merchandise information, tens of thousands of matching operations are required.

Therefore, classification of various products is required for quick retrieval of goods. In accordance with this demand, a search for a product classification is required to know what kind of product the product is in the image-based search

The present invention is designed to satisfy the above-mentioned needs. In order to recognize an object using an image of a product of interest at a predetermined terminal or a web site, the present invention primarily analyzes the type of the product of interest using the image minutia information A method and system for providing a product classification search and a shopping information service through object recognition that can additionally provide shopping information for a product detected and searched for a product or a similar product similar to a product of interest through histogram analysis within the type, The system has a purpose to provide.

According to a first aspect of the present invention, there is provided a service for providing goods classification search and shopping information through object recognition, comprising: an image acquisition unit for acquiring goods of interest and generating an image of the goods of interest, and transmitting the images through a wired / wireless communication network terminal; And storing the shopping information for the desired product. When receiving the input image to be searched from the terminal, the input image is analyzed to classify the same or similar goods of interest And an image recognition server for extracting shopping information on the identified interest goods and providing the extracted shopping information to the terminal.

According to a second aspect of the present invention, a goods classification search and shopping information providing server through object recognition is connected to a terminal through a wire / wireless communication network and provides shopping information on a desired product to the terminal, A commodity DB in which minutiae information for each commodity classification is classified and stored; A shopping information DB storing shopping information about the goods; An image recognition unit for extracting a feature point from the input image received from the terminal and comparing the extracted feature points with the feature point information stored in the product DB to recognize a product class of the same or the most similar product of interest; And a shopping information management unit for extracting shopping information mapped with the goods recognized by the image recognition unit from the shopping information DB and providing the extracted shopping information to the terminal.

According to a third aspect of the present invention, there is provided a method of providing a goods classification search and a shopping information providing service through object recognition, wherein a server and a terminal are connected to each other through a wire / wireless communication network to provide a shopping service for goods of interest a) generating and storing an image of the desired product when a desired item of interest is photographed through the terminal, and transmitting the image to the server via a wire / wireless communication network at the request of the user; (b) analyzing the input image received from the terminal to identify the same or the most similar items of interest and recognizing the goods classification thereof; And (c) the server extracting the shopping information on the item of interest identified in the step (b) and providing the shopping information to the terminal.

According to the present invention, when an object recognition is performed on an image basis, it is possible to know what type of an input image is to be searched, thereby improving the search ability of the corresponding product and increasing the reliability of the search.

Also, there is an effect that, by extracting the same or other similar goods as the input image in the product type to which the input image belongs, shopping information on the similar goods can be provided in the absence of the same product or in an additional aspect. Thereby, shopping purchasing can be activated.

In addition, it is possible to easily acquire an image of a desired commodity by capturing a commodity of interest that is found by chance during the in-line shopping or usually by using a terminal which can be easily provided by anyone and can check the commodity again at a later time.

1 is a configuration diagram illustrating a shopping information providing service system according to an embodiment of the present invention.
2 is a block diagram of a terminal according to an embodiment of the present invention.
3 is a configuration diagram of an image recognition server according to an embodiment of the present invention.
4 is a diagram illustrating a structure of a product DB in the image recognition server according to the embodiment of the present invention.
5 is a diagram illustrating a process of searching for a product classification based on minutia information in an image recognition server according to an embodiment of the present invention.
6 is a flowchart illustrating a method for providing a goods classification search and a shopping information providing service through object recognition according to an embodiment of the present invention.
7 is a detailed flowchart of Fig.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The configuration of the present invention and the operation and effect thereof will be clearly understood through the following detailed description. Before describing the present invention in detail, the same components are denoted by the same reference symbols as possible even if they are displayed on different drawings. In the case where it is judged that the gist of the present invention may be blurred to a known configuration, do.

The object recognition described in the embodiment of the present invention is broadly understood to recognize a photographic image photographed through a terminal as an image basis. In a small sense, the background image and the object are separated from the image generated through the photographing means of the terminal It can be interpreted as recognizing only an object. The embodiment of the present invention includes both of the meanings.

1 is a configuration diagram illustrating a shopping information providing service system according to an embodiment of the present invention.

Referring to FIG. 1, a shopping information service providing system according to an embodiment of the present invention includes a terminal 100, a wired / wireless communication network 200, and a mobile terminal 100, which are capable of searching for desired products (hereinafter, (300) for recognizing and searching an object based on the image received from the terminal (100) through the Internet network (400), and a shopping service of the interested goods registered in the image recognition server (300) And a shopping information server 500 that stores the shopping information.

The terminal 100 can be applied to any type of terminal if it is a terminal capable of transmitting and receiving data through the wired / wireless communication network 200 and acquiring images of actual goods. For example, a PC, a PDA, a digital camera, a web cam, a mobile communication terminal, and the like can be applied. When the terminal 100 is a mobile communication terminal, the wired / wireless communication network 200 may be a mobile communication network.

In particular, the terminal 100 according to the embodiment of the present invention creates and stores an interested product image by photographing an interested item that is found by accident during shopping or in accordance with a user's operation, When the user wants to manage the product of interest, the image of the product of interest is transmitted to the image recognition server 300 through the wired / wireless communication network 200 according to the user's selection.

To this end, the terminal 100 according to the embodiment of the present invention includes photographing means such as a camera for photographing a product and generating an image.

On the other hand, the image of a product of interest described below may be a photographic image directly photographing a product of interest using the terminal 100, but the present invention is not limited thereto. For example, the image may be a video obtained through a web site, .

The terminal 100 according to the embodiment of the present invention may store location information about a location where a desired product is photographed, time at which the interested product is photographed, that is, date and time information about date and time, And at least one of other information including additional information may be input and included in the interested merchandise image.

In addition, the terminal 100 according to the embodiment of the present invention does not receive the location information of the desired product from the user, and when a product of interest is photographed, the terminal 100 includes a global positioning system (GPS) It is possible to grasp the photographed position and add the positional information to the interested goods image.

In addition, the terminal 100 according to the embodiment of the present invention accesses the shopping information server 500 using a data communication service or an Internet service provided by the wire / wireless communication network 200, You can be provided with a purchase service.

In addition, the terminal 100 according to the embodiment of the present invention divides the image of the interested goods photographed through the photographing means into regions of a predetermined size, blurring the divided regions, and converts the divided images into a plurality of different images To perform image preprocessing. Therefore, a number of pre-processed image files are used for object recognition.

As described above, the most important reason for performing the image preprocessing process is that when the minutia information stored in the DB is different from the minutia information extracted from the image taken from the terminal 100 during the object recognition, information to be compared or mapped is different This is because the image-based search is not performed. For example, since the information stored in the DB is minutia information extracted from the reference image photographed with 2,000,000 pixels, the information extracted from the terminal 100 is minutia information extracted from the image of 5,000,000 pixels, A larger number of minutia information than information can be extracted. Therefore, in order to retrieve an optimal image in object recognition, a blurring process is required in advance of image preprocessing. The specific configuration of the terminal 100 for implementing this will be described later in detail with reference to FIG.

The wired / wireless communication network 200 is a network that provides wired / wireless communication with the terminal 100 to provide services. The wired / wireless communication network 200 may be a network for providing a variety of mobile communication services such as a voice call service and a data communication service to a mobile communication terminal when the terminal 100 is a mobile communication terminal.

In addition, the wired / wireless communication network 200 may include a WiFi network capable of providing high-speed Internet within a certain distance where the wireless access point (AP) is installed.

The wired / wireless communication network 200 according to an embodiment of the present invention receives an image of a product of interest from the terminal 100 and transmits the image to the image recognition server 300 .

In addition, when the wired / wireless communication network 200 according to the embodiment of the present invention is requested to transmit the shopping information for the interested goods from the image recognition server 300, the shopping information for the interested goods is received from the image recognition server 300, To the terminal (100).

For this purpose, the wired / wireless communication network 200 according to the embodiment of the present invention includes a short message service (SMS), a multimedia messaging service (MMS), a wireless Internet service, Server 300 so that video and shopping information about a product of interest can be transmitted to each other.

The Internet 400 allows the terminal 100 and the shopping information server 500 to communicate with each other to perform data communication with each other. The Internet 400 is a wired or wireless network that exists in the TCP / IP protocol and its upper layer, namely, Hypertext Transfer Protocol (HTTP), Telnet, File Transfer Protocol (FTP), Domain Name Server (DNS) Transfer Protocol, Simple Network Management Protocol (SNTP), Network File Service (NFS), and Network Information Service (NIS).

In the Internet 400, the wired / wireless communication network 200 may use the same network in the case of a wired / wireless Internet communication network.

The image recognition server 300 registers the minutia information of each item of interest in advance in order to recognize the object of the image of interest, and provides shopping information of interest goods registered in the database to the shopping information server 500 . The shopping information may include the lowest selling price, the highest selling price, the IP address of the online shopping mall site, and the like, as information necessary for shopping purchase of the interested merchandise.

In particular, the image recognition server 300 according to the embodiment of the present invention receives an image of a product of interest from the terminal 100 connected to the wired / wireless communication network 200 or the Internet network 400, Extracts minutiae points and compares them with minutiae information previously stored in the database. Analyzes the type of the interested merchandise and the interested merchandise through comparison / analysis, detects shopping information of the interested merchandise, and transmits the detected shopping information to the terminal 100 together with the merchandise type information.

The shopping information server 500 may be implemented as a typical network server to manage shopping items of various goods, in particular, interest goods registered in the image recognition server 300, and may operate an online shopping mall site for selling the goods of interest. In this case, the shopping information server 500 sells and delivers the purchased product in response to the purchase request of the terminal 100. [

The shopping information provision service system according to the embodiment of the present invention configured as described above identifies an interested item stored in an image through object recognition when a video of a desired item to be searched through the terminal 100 is input, Identify what type of product it is. In addition, shopping information on the same commodity as the image is additionally provided by the above-described grasp, and shopping information on the most similar commodity is provided when the same commodity is not present.

 Accordingly, the user can easily understand the shopping information about the commodity similar to the interested commodity, not only the interested commodity desired by the user, but also can activate the shopping purchase.

2 is a block diagram of a terminal according to an embodiment of the present invention.

The terminal 100 according to the embodiment of the present invention includes a camera 110 for photographing a product of interest and generating an image thereof, a preprocessor 120 for preprocessing an image of a product of interest, A memory 140 for storing an image of a desired product generated through the camera 110 and an image passed through the preprocessing unit 120, a memory 140 for storing a search request for a desired product, 140 to the wired / wireless communication network 200 (FIG. 1).

The camera 110 generates an image of a product of interest as a photographing device included in the terminal 100, and transmits the image to the preprocessing unit 120.

The preprocessing unit 120 includes an area dividing unit 122 dividing an image received through the camera 110 into a predetermined size area, and a blurring processing unit 122 dividing the divided image into a plurality of And an image converting unit 124 for converting the input image into an image. The preprocessing unit 120 performs a function of generating a plurality of images by correcting one image generated by the camera 110 for each region.

The blurring process is a method of checking the average value of the color and brightness of a plurality of pixels located around the pixels included in each region and adjusting the color and brightness of the pixels included in each region based on the average value to be.

Since the resolution of the image is changed according to the number of times of performing the blurring process and the blurring process is performed for each region, a plurality of images having different resolutions from one image can be generated.

3 is a configuration diagram of an image recognition server according to an embodiment of the present invention.

The image recognition server 300 according to an exemplary embodiment of the present invention analyzes an input image of a desired product received through the terminal 100 and identifies an interested product in the image, .

To this end, the image recognition server 300 includes an image recognition unit 310 for recognizing an object of interest and its type through object recognition from an input image, a product DB 320 for storing minutia information of each interested product for object recognition, A shopping information management unit 330 for managing shopping information of a desired product, and a shopping information DB 340 for storing shopping information of each desired product.

Specifically, the image recognition unit 310 compares the extracted feature points with the feature point information stored in the product DB 320 to search for the type of the product And a histogram analyzing unit 316 for applying an accurate search in retrieving the goods classification.

The feature point extraction unit 312 extracts an image pattern value from an input image using an edge algorithm and a grid algorithm. The former edge algorithm extracts the outer edge of the image and the latter grid algorithm uses the feature point in the corner of the image and the convex region. ) Is extracted. The edge algorithm and the grid algorithm are well known in the field of image technology, and the specific extraction method is based on known techniques.

In this manner, the feature point extracting unit 312 combines the two algorithms and accurately extracts feature points according to the pattern of the object.

The goods classification searching unit 314 searches the goods classification based on the minutia information extracted by the minutiae extracting unit 312. [ That is, a search is made as to whether the minutia information extracted by the minutia extraction unit 312 exists in minutia information previously stored in the commodity DB 320, and if so, which commodity classification is located. This is because the various items of interest are stored in the product DB 320 according to the product category, so that if the user knows only the location where the item of interest is stored, the product category can be known.

At this time, the goods classification searching unit 314 according to the embodiment of the present invention searches for using the minutia information, so that it is possible to retrieve several goods categories having minutia information of the input image. In other words, because the minutia information is similar to the minutia information if the shape or appearance of the object is similar even though the actual product itself is different, the minutia information can be retrieved up to the undesired product category. In the example shown in FIG. 5, when extracting minutia information of a mobile phone image to search for a product classification, it is shown that a DVD player, a washing machine, a refrigerator, and the like are searched for a similar item to minutia information of a mobile phone. Accordingly, the product category other than the product category to which the mobile phone belongs is searched up to another product category to which the DVD player, the washing machine, and the refrigerator belong, so that accurate product search can not be performed.

In this case, the image recognition unit 310 performs a detailed search using the histogram analysis unit 316. [

The histogram analyzer 316 compares images using a histogram algorithm when searching for an object. In this method, images are displayed as a color distribution graph, and then the graphs obtained from the respective images are compared with each other to determine similarity of the images by differences between the graphs. At this time, the histogram algorithm can obtain a color distribution diagram based on RGB color values of pixels existing in a specific region or one image.

For example, a number of pixels are arranged to represent a single eye, and these pixels have color values (hereinafter, referred to as pixel values) of R (red), G (green), and B . For example, one pixel value representing an eye has R, G, and B color values in order (98, 74, 47) and the neighboring pixels have color values of (194, 133, 112) The color values of other adjacent pixels may be (216, 175, 164). The histogram can be generated by mapping these pixel values to the 64 divided regions of the three-dimensional color space to obtain the frequencies and normalizing the sum to be 1. In this way, the histogram of the input image is obtained and compared with the histogram of the comparative image, so that the degree of similarity between images can be easily analyzed.

The commodity DB 320 stores the minutia information extracted from the image of the commodity of interest by the type of each commodity of interest. For example, if the product of interest is a computer, a mobile phone, or an MP3, the product classification is classified into a computer / electronic type. If the product of interest is a refrigerator, a washing machine, or a microwave, The product category can be classified as a fashion category. As shown in FIG. 4, each product category A (322), product category B (324), and product category C (326) may include a product name, feature information, classification, size, weight and other information for each product.

The shopping DB 340 may include a lowest selling price, a highest selling price, and an IP address of an online shopping mall site registered in the merchandise DB 320.

The shopping information management unit 330 periodically receives the shopping information on the interested goods from a plurality of shopping information servers (500 in FIG. 1) providing the shopping information of the interested goods, and stores and updates the shopping information in the shopping information DB 340 . Then, if it is identified as a particular item of interest through the image recognition unit 310, the shopping information matching with a particular item of interest is extracted from the shopping information DB 340 and provided to the corresponding terminal (100 of FIG. 1).

Therefore, in the embodiment of the present invention, the image recognition server 300 receives the image of the interested goods photographed in the terminal (100 in FIG. 1), searches for the same or similar image as the image, and provides detailed information. In particular, the present invention can search for a product category to which an image of a product of interest belongs and grasp it, and provide shopping information for a similar product in addition to the same product, thereby further activating a shopping service through object recognition.

6 and 7, a service method for providing product classification search and based shopping information using a shopping information service providing system based on the above configuration will be described.

First, the image recognition server 300 classifies and stores minutia information of interest goods to be registered in the commodity feature point DB, which is an internal database, according to the commodity classification, and receives shopping information about each commodity of interest from the shopping information server 500 And stores it in the information DB and updates it (S100, S110).

Thereafter, the user captures an item of interest to purchase the product during shopping or through the terminal 100, and stores an image of the product of interest (S120).

After that, the terminal 100 divides the input image of the stored interest goods into regions and performs preprocessing through blurring according to the divided regions, thereby generating pixels as different images (S130).

Thereafter, when the terminal 100 requests a search for the item of interest by the user's request, the terminal 100 transmits the corresponding image to the image recognition server 300 through the wired / wireless communication network (S140, S150).

The image recognition server 300 analyzes the input image received from the terminal 100 and recognizes the product classification of the product of interest (S160, S170).

7, minutiae points are extracted from the input image received from the terminal 100 using the edge algorithm and the grid algorithm (S161), and the minutiae points extracted are compared with the minutiae points stored in the database of the image recognition server 300 (S162).

As a result of the analysis, the product classification group including the minutia information of the input image received from the terminal 100 is extracted (S163). The product tax group thus extracted may be a genuine product classification including minutia information of the input image received from the terminal 100, but it may include an undesired product classification depending on the shape and appearance of the product.

Accordingly, it is confirmed whether the extracted commodity taxa are plural, that is, two or more (S164). If there are two or more of them, it means that there are several goods having similar minutia information. In this case, the interested goods which are the same as or most similar to the input image are recognized through the histogram analysis, and the goods classification of the interested goods is extracted and recognized (S165, S166).

If the extracted product classifications are single, but not multiple, the extracted information is recognized as corresponding product classifications and products (S166).

Then, as shown in FIG. 6, the image recognition server 300 notifies the product classification information to the terminal 100 (S180).

In step S190, the image recognition server 300 searches for and extracts shopping information on the recognized target merchandise through the image analysis process S160.

Then, the extracted shopping information is transmitted to the corresponding terminal 100 (S200).

Then, the terminal 100 grasps shopping information, directly links to the shopping information server 500, which is an online shopping mall site, and receives purchase related services from the shopping information server 500 (S210, S220). This allows actual purchases to be made at the shopping mall.

The foregoing description is merely illustrative of the present invention, and various modifications may be made by those skilled in the art without departing from the spirit of the present invention. Therefore, the embodiments disclosed in the specification of the present invention do not limit the present invention. The scope of the present invention should be construed according to the following claims, and all the techniques within the scope of equivalents should be construed as being included in the scope of the present invention.

In the conventional online shopping information providing service, shopping information is provided through a search service in order to obtain accurate information on a desired item of interest, such as a product name or a product code of a product of interest. However, It is possible to acquire images easily from anywhere, and when the object is recognized on the server, it is possible to recognize precisely by recognizing the type of the interested product through the feature point base and the histogram analysis, and the reliability thereof can be increased. In addition, shopping information on a product of interest can be provided together to facilitate convenient shopping, thereby enabling the online shopping purchase service business through the present terminal to be more active.

100: terminal 200: wired / wireless communication network
300: image recognition server 400: internet network
500: shopping information server
110: camera 120: preprocessing unit
122: area division unit 124: image conversion unit
130: Display 140: Memory
150: transmitting unit 310:
312: feature point extracting unit 314: product classification searching unit
316: Histogram analysis unit 320: Product DB
330: shopping information management unit 340: shopping information DB

Claims (12)

delete delete A server connected to a predetermined terminal through a wired / wireless communication network to provide shopping information on a desired product to the terminal,
A commodity DB in which minutia information about the commodity is classified and stored according to commodity classification;
Performing a first search process of extracting feature points from the input image received from the terminal and comparing the extracted feature points with the feature point information stored in the product DB to search for a product class having the extracted feature points, And performing a detailed image analysis based on a histogram of the input image to recognize a product of interest that is the same as or the most similar to the input image when at least two items of the product classification are searched as a result of the search, An image recognition unit for performing a second search process of searching for a specific product category including the desired product during classification; And
And a shopping information management unit for providing the shopping information mapped with the interest goods recognized by the image recognition unit to the terminal,
And a shopping information providing server for providing goods information to the shopping center.
The method of claim 3,
Wherein the image recognizing unit comprises:
Wherein feature points are extracted from an outline, a corner, and a convex region of an object using an edge algorithm and a grid algorithm when extracting feature points from the input image. And a shopping information providing server.
The method of claim 3,
And a shopping information DB storing shopping information about the goods. The server according to claim 1,
The method of claim 3,
Wherein the product DB includes information on a product name, feature information, product classification, size, and weight of the product.
6. The method of claim 5,
Wherein the shopping information includes at least one of a selling price, a selling point, a minimum price, a maximum price, and an IP address of an online shopping mall site for the goods.
The method of claim 3,
The shopping information management unit
Wherein the shopping information is periodically received and updated in the shopping information DB. ≪ RTI ID = 0.0 > 15. < / RTI >
The method of claim 3,
The shopping information management unit
And the shopping information of the goods similar to the interest goods other than the shopping information of the interest goods recognized by the image recognition unit can be extracted and additionally provided.
A method for providing a shopping service for a commodity of interest to each other by connecting a server and a terminal through a wire / wireless communication network,
(a) performing a first search process of extracting feature points from the input image received from the terminal, comparing the extracted feature points with feature point information stored in the product DB, and searching for a product class having the extracted feature points;
(b) performing a detailed image analysis on the basis of the histogram of the input image if at least two items of the goods classification are searched as a result of performing the first searching process to recognize an item of interest that is the same as or most similar to the input image Performing a second search process of searching for a specific product category including the interest goods among the at least two goods categories; And
(c) providing shopping information mapped with the desired product to the terminal
The method according to claim 1, wherein the service information includes at least one of a service name,
delete 11. The method of claim 10,
The step (c)
Wherein the shopping information of a product similar to the desired product is extracted and additionally provided in addition to the shopping information of the desired product.

Images