KR20140121156A - Search Method by Object Recognition on Image and Search Server thereof - Google Patents

Abstract

Disclosed is a search method by recognizing an object in an image. The search method of the present invention is able to recognize an object by separating the object which is selected by a user from an image and is also able to search content from a database based on the recognized image part. For the forementioned, the present invention proposes a method to extract an object in a method of grouping pixels having high coherence after calculating coherence between pixels based on color similarity, a pixel distance, etc. Furthermore, the method of the present invention is able to search content which has same or similar specific point information as an identifier from a database, etc. and provide the content by extracting the specific point information of an object if the corresponding object to be searched is selected from extracted objects.

Description

[0001] The present invention relates to a search method by object recognition on an image and a search server by the search method,

The present invention relates to a retrieval method and a retrieval method through object recognition on an image, in which an object selected by a user is separated and recognized in one image and a content on the database is retrieved based on the recognized image portion Search server.

With the development of computer technology and the Internet, too many contents are accumulated and stored. Since such information is not structured in a systematic manner in accordance with a unified standard in an integrated system, the importance of searching is also increasing, and various search techniques are being developed.

A known search method is to use 'text' as the search term. It is a method of presenting contents containing the same or similar text as a search word. For example, in the case of searching an image, a search word is input in a text format, and an image combined with a text identical or similar to the text in the images is extracted and presented.

However, there is a certain limit to the search by text. For example, as shown in Fig. 3 (a), let's say that a user having a picture p1 of a certain clothes wants to search for the clothes on the Internet. The only information the user can present is images. More precisely, the real information is only the image p2 of the specific object (clothes) contained in the photographic image p1. In this case, only the currently known search method can not use the image itself, and the user interprets the image as text and searches for text.

As described above, there is no effective method for searching images using an image as a search word. The image search is performed by recognizing an image to be a search object instead of a text search word, extracting feature data from the image, and this process is not easy.

[Related Technical Literature]

1. Korean Patent Publication No. 2002-0075600 [Image Retrieval Method and Apparatus]

The present invention relates to a method for improving the reliability of a search result by performing a secondary search after a user makes a similar selection by presenting a primary search result to a user in order to compensate for an incorrect result of an image search. That perspective is different.

It is an object of the present invention to provide a search method and an image search method for recognizing an object selected by a user in an image and capable of searching contents in a database based on the recognized image portion, And a search server according to the method.

According to another aspect of the present invention, there is provided a method of searching through an object on an image, the method including: receiving a basic image including an image of an object to be searched, Receiving a selection command from the user to select a target object to be searched from the base image and recognizing the target object; Extracting a feature point from the target object and generating feature point information on the geometric arrangement pattern of the feature point; Retrieving a content having the same or similar minutia information as the generated minutia information from the database; And the similarity retrieval section providing the retrieved contents to the user.

According to an embodiment of the present invention, the step of recognizing a target object of the object extracting unit includes the steps of (1) extracting at least one object included in the base image, (2) selecting the target object among the extracted at least one object, Receiving a selection command from the user, (3) displaying the target object,

According to another embodiment, there is provided an image processing method comprising the steps of: (1) receiving a selection command for selecting an image area of the target object from a basic image that has not been extracted, from the user; (2) And (3) display the extracted object as the target object.

In addition, the step of extracting the object may include the steps of: (a) selecting a plurality of pixel pairs that can be combined according to a predetermined reference on the base image, and determining a binding force between the pixel pairs based on the color similarity of the pixel pair and the inter- ; (b) filtering, among the plurality of pixel pairs, a pair of pixels having the calculated binding force equal to or greater than a preset reference value, and ignoring the others; (c) interconnecting and grouping the filtered pixel pairs; And (d) extracting, from the group generated in the grouping step, a group having a reference size or more as the object, and ignoring the others. Here, the 'color similarity' represents the similarity of color values of two contrasting pixels in the pixel pair, the 'inter-pixel distance' represents a relative distance between two pixels for obtaining a binding force,

으로부터 구할 수 있다.The binding force is expressed by the following equation,

.

According to another embodiment of the present invention, a search server connected to a database and performing a search through object recognition on an image may include an object extraction unit and a similarity search unit.

A retrieval method based on object recognition on an image of the present invention recognizes an object region selected by a user in the presented image and extracts feature data of the recognized object region to estimate information about the object, Search.

The search method of the present invention

1 shows a search system according to an embodiment of the present invention,
FIG. 2 is a flow chart for explaining a search method through object recognition on an image of the present invention;
3 is a diagram showing a basic image and a target object,
Figure 4 is a flow chart provided in the description of the object extraction method of the present invention, and
5 is a diagram provided in the description of the object extraction method of FIG.

BEST MODE FOR CARRYING OUT THE INVENTION The present invention will be described in more detail with reference to the drawings.

The search method of the present invention may operate based on various hardware configurations, but one of them is based on the Internet as shown in FIG. In this case, the search method of the present invention is implemented in the search system 100 including the database 110, the search server 130, and the terminal 150 to be connected to the search server 130, ).

Various contents (e.g., images) are stored in the database 110, and each of the contents is identified by its own unique feature point information to be described below. In other words, the search server 130 searches for the desired content by querying the 'feature information'.

The search server 130 includes an object extraction unit 131 and a similarity search unit 133 for the image-based search of the present invention. The operation of the object extracting unit 131 and the similarity search unit 133 will be described below again.

Of course, the terminal 150 may be a mobile terminal such as a smart phone in addition to a normal personal computer. When the terminal 150 is a smart phone, the terminal 150 may be provided with an application program for providing a search service in connection with the search server 130. [

The search method in the system 100 is performed by the search server 130 recognizing the target object to be searched from the basic image presented by the user and searching the database 110 for a content similar to the recognized target object do.

Hereinafter, the search method of the present invention will be described in more detail with reference to FIG. First, for convenience of explanation, the following matters are assumed. Suppose that the user desires to search the Internet for clothes (or information about it) that looks similar to the image p2 captured on the image p1 shown in Fig. 3 (a).

<Basic image presentation: S201>

The basic image is an image (p1) of Fig. 3 (a), which is an image including an object to be searched as a result.

The user may access the search server 130 through the terminal 150 and then provide a basic image to the search server 130 in various ways. (1) a method of directly inputting a URL of an image existing in a specific web page of the Internet 170 (2) a method of directly capturing the URL of the image present in the terminal 150, Method, and the like.

&Lt; S203 &gt; At least one object (Object) is extracted from the &

The object extracting unit 131 of the search server 130 extracts an object to be recognized from the basic image provided by the terminal 150. For example, the objects p2 and p3 are extracted from the base image p1 in Fig. 3A. The object extraction method based on the operation of the object extraction unit 131 will be described below again.

The object extracting unit 131 extracts the two objects p2 and p3 from the base image p1 shown in FIG. 3A and outputs the objects p2 and p3 to the user as shown in FIG. It is possible to separately display and distinguish the outline of the image.

The method of displaying the object extracted by the object extracting unit 131 to the user is not limited to the method of separately displaying the outline as shown in FIG. 3 (b), the method of displaying only the extracted object in color, can do.

<Selection of Search Object: S205>

If the object extracting unit 131 extracts the object from the basic image p1 and displays it to the user in step S203, the similarity search unit 133 may input a 'selection command' for selecting one of the displayed at least one object from the user Receive. Hereinafter, for convenience of description, an object selected by the user, that is, an object to be searched is referred to as a 'target object'. It is also assumed that the user has selected the object p2 as the target object in FIG. 3 (b).

<Image based search: S207>

The similarity search unit 133 generates 'minutia information' from the target object p2 selected by the user and searches the database 110 for a content having minutia information identical or similar to the extracted minutia information. Herein, the minutia information is information on the geometrical arrangement pattern (shape) of the minutiae points of the target object p3, and color information may also be included in the minutia information according to the embodiment.

To this end, the similarity search unit 133 first extracts feature points from the target object p2, and then collects geometric arrangement pattern information of the detected feature points, that is, 'feature point information'. Extracting feature points and generating feature point information will be possible in various ways. For example, in the target object p2 in the base image p1, a point having unique (much different) color characteristics from surrounding pixels can be detected as a feature point.

When the 'similarity search unit 133 generates' minutia information' from the target object p2, the similarity searching unit 133 searches the database 110 for a content having minutia information identical or similar to the generated minutia information as an identifier.

As described above, a plurality of contents are already stored in the database 110, and the identifiers are unique minutia information. If the stored content is an image, the minutia information to be an identifier may be minutia information of one of the objects included in the image.

In general, all contents having minutia information within a certain range of similarity can be searched. Since the minutia information is related to the geometric arrangement pattern of the minutiae (including color according to the embodiment), it generally represents the morphological characteristic of the target object p3. Therefore, the search of the similarity search unit 133 is the same as the search of an image including an object having a similar shape.

<Search result provided: S209>

The similarity search unit 133 transmits the search result of the database 110 to the terminal 150 or provides the result to the user through a web page or the like.

Returning to the example of FIG. 3, the database 110 stores images of clothes sold in various shopping malls, seller information and price information for selling the clothes together. The similarity search unit 133 searches clothes having similar morphological characteristics to the target object p2 selected by the user in (b) of FIG. 3, and provides information about sellers and price information together .

Various embodiments are possible depending on the data mapped to minutia information. For example, when sound source data or voice data is mapped to the corresponding minutiae in the database 110, the similarity search unit 133 extracts minutia information or voice data having similar minutia information within the critical range as identifiers And may provide a service to reproduce.

In this way, the image-based search of the present invention is performed.

In the search method of FIG. 2, the object extracting unit 131 extracts all the objects in the basic image and displays the objects, and then confirms the target object to be searched as a selection command. However, the method is not limited thereto. For example, the object extracting unit 131 displays the basic image p1, which is not preceded by the object extraction, to the user as it is, receives a selection command for selecting a part of the object that the user desires, As a target object.

Furthermore, the search method of the present invention does not necessarily have to be based on the system 100 of FIG. For example, the present invention may be presented as a method for one of devices including a configuration corresponding to the object extraction unit 131 and the similarity search unit 133 to search for one of information stored in the device.

Hereinafter, an object extraction method of the object extraction unit 131 will be described with reference to FIGS.

The object extracting unit 131 calculates the 'binding force' between the pixels and extracts the objects by grouping the pixels having high bonding power. Here, the bonding force represents a degree of belonging to the same object as a relative value with respect to the degree of coupling between two contrasting pixels. The object extracting unit 131 determines that the two pixels have similar colors and the closer the distance therebetween, the higher the binding force, that is, belongs to the same object. Finally, the object extracting unit 131 recognizes only the objects having a size of the combined group equal to or larger than a predetermined size, and ignores objects smaller than the predetermined size as objects.

<Calculation of Bonding Force between Pixels: S401>

Referring to FIG. 4, the object extracting unit 131 calculates a binding force for all of a plurality of virtual pixel pairs that can be combined with two pixels selected as an arbitrary reference in all pixels in the base image p1. Here, the combination of the pixel pairs may include (1) all cases where the pixels are not overlapped, and (2) the case where the distance between the two pixels is within a certain range may not be overlapped. The bonding force can be obtained by using the following equation (1).

Here, 'color similarity' represents a degree of similarity between the color values of two pixels included in the pair of contrasting pixels, and 'distance between pixels' represents a relative distance between two pixels included in the pair of pixels. Therefore, the binding force is a value for the relationship of two pixels, and the color of two pixels is similar and the closer the distance between the two pixels is, the larger the value is. Thus, the binding force can be defined as a relative value relative to the degree of coupling between two pixels included in a pixel pair, proportional to the color similarity and inversely proportional to the distance between the pixels.

&Lt; Extracting pixel pairs having effective binding force: S403 >

The object extracting unit 131 filters and extracts the pixel pairs having the binding force calculated in step S401 by a predetermined reference value or more and ignores the pixel pairs having a binding force smaller than the predetermined reference value. Here, the reference value can be obtained experimentally.

Through this process, pixel pairs with greatly different colors or with their distances far removed are excluded, leaving only a pair of stronger pixels.

<Grouping based on pixel pairs: S405>

The object extracting unit 131 performs grouping to connect the stronger pixel pairs having passed the step S403. Therefore, grouping is performed on a pair of pixels having a certain level of cohesion. The grouping is performed by marking that two pixels in the cohesive strong pixel pair that passed step S403 are included in the same object.

For example, suppose that FIG. 5 is an 8x8 pixel sized image. Since the image shown in FIG. 5 is too small, it is assumed that the binding force is obtained only for a pair of pixels directly adjacent to each other according to (2) of the pixel pair combination described above. For example, the measurement of the bonding force of the pixel x2 proceeds only with eight pixels (x1, x3, x4, x7, x8, x14, x15, x16) which are in direct contact with each other and not with the x5 or x17 will be.

(X1, x2), (x1, x3), (x3, x4), (x3, x5), (x4, x6) (x12, 13) and the like are a pair of pixels having a strong coupling force, and the pair of pixels (x14, x15) is also assumed to have a coupling force equal to or greater than a reference value. The pixel pairs (x2, x7), (x7, x9), (x4, x8), (x8, x12) ) Is also assumed to be a weak pixel pair.

Because pixels x1 and x2 belong to the same object and pixels x1 and x3 belong to the same object, pixel x1, x2, and x3 are grouped into the same group by the pixel pair (x1, x2), (x1, x3). (X3, x4), (x3, x5) having a strong coupling force with any one of these pixels x1, x2, and x3 are grouped into these groups, and x6, which has a strong coupling force with the pixels x4 and x5, . However, the pixels x7, x8, etc. do not have the coupling force with any of the pixels x1 to x6, and therefore they can not be combined into the same group. Even though the pixel x9 has high color similarity with the pixels of these groups, the bonding force is weak due to the distance between the pixels, so that the same group can not be formed and a separate group is formed. If pixels x7 and x8 have strong cohesion, pixels x7 and x8 will also be grouped into one group.

Groups formed through this grouping process become objects.

<Exclusion of small group or background: S407>

The object extracting unit 131 may determine that the group having the size smaller than the predetermined size among the groups generated through the step S405 does not recognize the object as noise and ignores it as noise. For example, even though pixels x7 and x8 have strong cohesion, pixels x7 and x8 are ignored instead of being recognized as a single object.

Referring to FIG. 5, the object extracting unit 131 extracts and recognizes all four groups, that is, four objects, from FIG. 5 through such a process. 3 (b) corresponds to an example in which the shoe located at the right side of the basic image p1 is not recognized as an object because its size is small and is ignored. If the reference size is reduced, the shoe can be recognized as an object.

Also, the object extracting unit 131 does not recognize the background as an object but ignores it. A method of recognizing a background in an image can be performed by a method known in the art. For example, as in the case of Fig. 3, the portion extending from the edge of the image is recognized as the background. The object p2 in FIG. 3 shown in black and white image is a blue series and the object p3 is a white series, and is recognized as a separate object. However, the remaining part is treated as the background of the brown part extending from the edge of the basic image (p1).

In this way, a method of extracting an object from the basic image of the present invention is performed.

<Other Embodiments>

According to another embodiment of the present invention, in the case where the content stored in the database is an image, in addition to the minutia information, a basic image photographing location, an outline form of the target object, and color information, contrast information, .

Accordingly, when the similarity search unit 133 searches the database 110 in step S207, the basic image p1 is searched using both the location information on which the basic image p1 was shot, the outline form information, and the color information in addition to the minutia information The reliability of the search can be increased.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the invention as defined by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

Claims (10)

Receiving a basic image including an image of an object to be searched by an object extraction unit from a user;
Receiving a selection command from the user to select a target object to be searched from the base image and recognizing the target object;
Extracting a feature point from the target object and generating feature point information on the geometric arrangement pattern of the feature point;
Retrieving a content having the same or similar minutia information as the generated minutia information from the database; And
Wherein the similarity retrieving unit provides the retrieved contents to the user.
The method according to claim 1,
Wherein the step of recognizing a target object of the object extracting unit comprises:
(1) extracting at least one object included in the basic image;
(2) receiving a selection command from the user to select the target object from among the extracted at least one object; And
(3) a display step of displaying the target object.
The method according to claim 1,
Wherein the step of recognizing a target object of the object extracting unit comprises:
(1) receiving a selection command from the user to select an image area of the target object in the basic image;
(2) extracting an object including the selected region from the base image; And
(3) a display step of displaying the extracted object as the target object.
The method according to claim 2 or 3,
In the object extracting step,
(a) selecting a plurality of pixel pairs that can be combined according to a predetermined reference on the basic image, and calculating a binding force between the pixel pairs based on the color similarity of the pixel pairs and the inter-pixel distance;
(b) filtering, among the plurality of pixel pairs, a pair of pixels having the calculated binding force equal to or greater than a preset reference value, and ignoring the others;
(c) interconnecting and grouping the filtered pixel pairs; And
(d) extracting, from the group generated in the step (c), a group having a reference size or more as the object and ignoring the rest,
The 'color similarity' represents the similarity of color values of two contrasting pixels in the pixel pair, and the 'inter-pixel distance' represents a relative distance between two pixels for obtaining a binding force,
The binding force is expressed by the following equation,

Wherein the search is performed on the image.
5. The method of claim 4,
Wherein the contents stored in the database include seller information or price information together with an image of a specific article,
In providing the retrieved content to the user,
Wherein the similarity search unit searches the database for a content having the same or similar minutia information as the generated minutia information as an identifier and provides the user with seller information or price information together with the image of the searched product A method of searching through object recognition on a computer.
1. A search server connected to a database and performing a search through object recognition on an image,
An object extraction unit for receiving a selection instruction from a user and receiving a basic image including an image of an object to be searched from a user and selecting a target object to be searched from the basic image and recognizing the target object; And
And a similarity search unit for extracting a feature point from a target object recognized by the object extraction unit and generating minutia information that is information on a geometric arrangement pattern of the minutiae to search for a content stored in the database,
Wherein the similarity retrieving unit retrieves a content having the same or similar minutia information as the generated minutia information from the database as an identifier and then provides the retrieved content to the user. Perform the search server.
The method according to claim 6,
The object extracting unit extracts,
Wherein the display unit displays the target object by extracting at least one object included in the base image and displaying the selected object to the user and receiving a selection command from the user to select the target object from the extracted at least one object, A search server that performs a search through object recognition on an image.
The method according to claim 6,
The object extracting unit extracts,
A selection command for selecting an image region of the target object from the basic image is first received from the user and an object including the selected region is extracted from the base image to display the extracted object as the target object A search server for performing search through object recognition on the image.
9. The method according to claim 7 or 8,
Wherein the object extracting unit extracts an object,
Selecting a plurality of pixel pairs that can be combined according to a predetermined reference on the basic image, calculating a binding force between the pixel pairs based on the color similarity of the pixel pairs and the distance between the pixels, Extracting the object by filtering pixel pairs having a binding force equal to or greater than a predetermined reference value and ignoring the remainder,
A group less than a reference size among the groups generated in the grouping process is ignored without extracting the object,
The 'color similarity' represents the similarity of color values of two contrasting pixels in the pixel pair, and the 'inter-pixel distance' represents a relative distance between two pixels for obtaining a binding force,
The binding force is expressed by the following equation,

Wherein the search is performed through object recognition on the image.
10. The method of claim 9,
Wherein the contents stored in the database include seller information or price information together with an image of a specific article,
Wherein the similarity search unit retrieves a content having the same or similar minutia information as the generated minutia information from the database as an identifier and provides seller information or price information together with an image of the retrieved article to the user A search server that performs a search through object recognition on an image.

Images