KR20140066476A - Method and apparatus for searching information based on image - Google Patents

Abstract

The present invention relates to a method for searching information based on images, which is suitable to provide an information retrieval service based on images of photos taken by using a mobile terminal with a photo function. To this end, the present invention, unlike an existing method for providing an information retrieval service for a mobile terminal in a manner of choosing a region and inputting a search word, extracts a region name by performing mapping among GPS information, an address, and a region which are relative to a place where photos of a specific structure transmitted from the mobile terminal are taken, recognizes a character string indicating a specific name from images of the photos, makes a query by combining the extracted region name with the recognized character string, acquires search information related to the specific structure through a retrieval based on the made query, and transmits the acquired search information to the mobile terminal, thereby minimizing a user′s operation for information retrieval on the mobile terminal and thus improving ease of use for information retrieval of mobile terminal users.

Description

[0001] METHOD AND APPARATUS FOR SEARCHING INFORMATION BASED ON IMAGE [0002]

The present invention relates to a technique for providing an information search service to a portable terminal or the like, and more particularly to a technique for providing an information search service based on images of photographs taken using a portable terminal having a photographing function, A search method and a device therefor.

In general, portable terminals such as mobile phones, smart phones, smart pads, PMPs, and template PCs are equipped with GPS functions so that the user can access his / her location or desired location through a wireless network anywhere through a wireless network Or search for desired information.

For example, when it is desired to search related information about a product or service that is sold or served in a specific store after passing through a road, a user accesses a search server through a portable terminal, selects a desired region, Product name, service name, etc.), and the like, and can receive related information transmitted by the search server in response to the request.

Here, the related information that the user wants to know may be, for example, a commodity price, a service price, a discount status and a history, a discount card type and details.

Accordingly, the users of the portable terminal can receive the search service for the information desired by the user through the connection to the search server and the input of the search word anytime and anywhere.

Korean Patent Publication No. 2011-86505 (Published on July 28, 2011)

However, the conventional method of providing an information search service to a mobile terminal by selecting a region and inputting a search word involves a problem that requires a lot of user operations such as region selection, input of a search word, etc. for an information search request. This is a factor that deteriorates the usability of use.

According to one aspect of the present invention, there is provided a method of searching for a specific structure, comprising the steps of: mapping a GPS image, an address, and an area of a photographing location of the photograph, Recognizing a character string representing a specific name from the image; creating a query term by combining the extracted local name and the recognized character string; acquiring related search information of the specific structure through searching based on the created query term; And transmitting the acquired related search information to the mobile terminal.

According to another aspect of the present invention, there is provided an information processing apparatus including an interface block for receiving an image of a photograph for a specific structure transmitted from a portable terminal, and an interface block for performing mapping between the GPS information, A character string generation block for recognizing a character string representing a specific name from the image; a query word generation block for generating a query word by combining the extracted local name and the recognized character string; And an information search execution block for acquiring related search information of the specific structure through a search and providing the acquired related search information to the mobile terminal.

According to the present invention, when a portable terminal transmits a photographic image of a specific structure, an information search server extracts a region name and a character string from an image, and associates relevant information about a specific structure with a query term created by combining the extracted region name and character string The present invention provides a portable terminal that searches for information, thereby minimizing user operations for searching for information in the portable terminal, thereby further enhancing the usability of information search of the portable terminal users.

1 is a block diagram of an information search service system suitable for applying an image-based information search apparatus according to the present invention;
FIG. 2 is a block diagram of an image-based information search apparatus according to an embodiment of the present invention;
3 is a flowchart illustrating a main process of providing an image-based information search service to a mobile terminal according to an embodiment of the present invention.

The technical point of the present invention is that unlike the conventional method of providing an information search service to a mobile terminal by selecting a region and inputting a search word, Mapping between GPS information, address, and region is performed to extract a local name, a character string indicating a specific name is recognized from the image of the photograph, a query is made by combining the extracted local name and the recognized character string, The present invention obtains related search information of a specific structure and transmits the acquired search information to a portable terminal. Thus, the present invention can effectively solve the problems in the conventional system by realizing the objective through such technical means.

In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. It is to be understood that the following terms are defined in consideration of the functions of the present invention, and may be changed according to intentions or customs of a user, an operator, and the like. Therefore, the definition should be based on the technical idea described throughout this specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram of an information search service system suitable for applying an image-based information search apparatus according to the present invention. The information search service system includes a portable terminal group 102 composed of a plurality of portable terminals 102/1 - 102 / And an information search server 106 and the like.

1, each of the portable terminals 102/1 to 102 / n in the portable terminal group 102 includes a wireless portable terminal 102/1 to 102 / n used by users who desire to receive an image- Each of the portable terminals 102/1 - 102 / n is connected to a network 104 (e.g., a mobile phone, a PMP, a smart phone, a smart pad, a tablet PC, a netbook, The information retrieval server 106 can be provided with an image-based information retrieval service or the like which is provided through the network 104 according to the present invention. Here, it is assumed that each of the portable terminals 102/1 - 102 / n includes a camera function, a GPS function, a wireless Internet function, and the like.

The network 104 may be any one of a second generation (2G) communication network, a third generation (3G) communication network, a fourth generation (4G) communication network and a fifth generation And a wired communication network including a wireless communication network and a wired Internet. That is, the network 102/1 - 102 / n may provide a wireless communication path between each portable terminal and a wired communication path with the information search server 106.

That is, each of the portable terminals 102/1 - 102 / n transmits an image (image including GPS information) of a specific structure photographed through a camera function to the information search server 106 via the network 104 And to output (display) related search information transmitted from the search server 106. The search server 106 may search for information related to the search result,

Here, the specific structure may be an offline shop (or a shop) having all kinds of commercial activities such as a restaurant, a movie theater, a bookstore, a mart, a wholesaler, etc., and the GPS information may be information contained in a photograph, (GPS coordinates) of the portable terminal provided by the terminal with the photograph.

The related search information provided from the search server 106 to the portable terminal includes related search information related to an offline shop (store) for which a user of the portable terminal has requested a search, such as a product type and price, a service type and price, , Discount card type and details, and the like may be optionally included.

Next, the information search server 106 refers to an information search apparatus that provides an image-based information search service according to the present invention to each portable terminal (portable terminal users) according to the present invention. The information search server 106 may be configured, for example, as shown in FIG.

FIG. 2 is a block diagram of an image-based information search apparatus according to an embodiment of the present invention. The interface block 202 includes an area name extraction block 204, a character string generation block 206, a query word generation block 208, An information search execution block 210, and the like. The area name extraction block 204 may include a GPS-address converter 2041, a GPS-address mapping DB 2042, an address-area converter 2043 and an address-area mapping DB 2044, The character string generation block 206 may include a character inference unit 2061, a character DB 2062, an image extractor 2063, an image DB 2064, and a character determiner 2065.

Referring to FIG. 2, the interface block 202 includes an interface block 202, which stores a specific structure (for example, a restaurant, a movie theater, a bookstore, a mart, a wholesale store, Or the store), and transmits the image information to the area name extraction block 204 and / or the character string generation block 206. The information search execution block 210 extracts relevant search information A service type and price, a service type and price, a discount status and details, a discount card type and details, etc.) are delivered to the corresponding portable terminal that requested the information search service.

Next, the GPS-address converter 2041 in the area name extracting block 204 extracts GPS information from the image information of the photograph of the specific structure transmitted from the interface block 202, and outputs the extracted GPS information and GPS- Mapping the plurality of pieces of address information stored in the address mapping DB 2042 to convert (mapping) the GPS information into address information, and the like. The generated address information is converted into address- 2043). Here, the GPS information may be information included in the photograph, or may be position information (GPS coordinates) of the portable terminal provided by the portable terminal with the photograph.

To this end, the GPS-address mapping DB 2042 stores a plurality of address information corresponding to a plurality of GPS information.

The address-to-local converter 2043 converts the address information into a local name through mapping between the address information transmitted from the GPS-address converter 2041 and the regions stored in the address-area mapping DB 2044 And the generated region name is transferred to the query term generation block 208. [

For example, assuming that the address corresponding to the GPS information extracted from the image information of the photograph is "Fuyang-dong, Gwangjin-gu, Gwangjin-gu, Seoul," the area name may be matched as " To this end, the address-area mapping DB 2044 stores a plurality of area name information corresponding to a plurality of address information.

Next, the character inference unit 2061 in the character string generation block 206 analyzes the image to extract character information indicating a specific name (e.g., shop name, shop name, name, etc.) (E.g., a pseudo-string comparison algorithm) between a plurality of characters stored in the character string memory 2062, wherein the inferred candidate characters are transmitted to the string decider 2065 .

For this purpose, a plurality of characters are stored in the character DB 2062. The characters stored in the character DB 2062 may be, for example, a shop name, a shop name, a name, and the like.

Next, the image extractor 2063 can provide a function of extracting candidate images through calculation of the content-based similarity between a plurality of image information stored in the image and image DB 2064. Here, And transmitted to the string decider 2065.

To this end, a plurality of pieces of image information are stored in the image DB 2064. The image information stored in the image DB 2064 may be, for example, a shop image, a shop image, and the like.

The character determiner 2065 compares the size of the similarity between the candidate characters transmitted when the candidate characters deduced from the character inferring unit 2061 are transmitted, and outputs the candidate character having the relatively largest similarity among the candidate characters as a character string And determines the candidate image having the largest relative similarity among the candidate images as the mapped image by comparing the similarity sizes of the candidate images transferred when the candidate images extracted from the image extractor 2063 are transferred, And determining the character corresponding to the character string as a character string. Here, the determined character string (e.g., store name, store name, reciprocal name, etc.) is transmitted to the query language generation block 208.

In the present embodiment, the character inference method (image inference method) and the image inference method (image inference method) are concurrently explained. However, the present invention is not necessarily limited to this, It is needless to say that it is possible to apply only the image extraction technique alone, or to apply the image extraction technique when the character indicating the specific name is not extracted in the character inference.

Next, the query term generation block 208 may provide a function of automatically generating a query term by combining the extracted local name transmitted from the address-local converter 2043 and a character string transmitted from the string decider 2065. [ For example, assuming that the name of the region is "genius" and the string is "ant species ", the query term generation block 208 automatically generates a query term" Lt; / RTI >

Finally, the information search execution block 210 searches for a specific structure (e.g., a restaurant, a movie theater, a bookstore, a mart, a wholesale store, or the like) transmitted from the portable terminal through a search based on a query term transmitted from the query term generation block 208 And an offline shop (or a store) that performs commercial activities of the user terminal, etc., and provides the acquired related search information to the portable terminal through the interface block 202. Here, the related search information may be, for example, a product type and a price, a service type and price, a discount status and details, a discount card type, and a description.

The search in the information search execution block 210 based on the query term can be executed through searching of the external web or through a search into a search system specialized for a specific structure.

Next, a series of processes for providing an image-based information search service to users of the mobile terminal using the information search apparatus of the present invention having the above-described configuration will be described with reference mainly to FIG.

3 is a flowchart illustrating a main process of providing an image-based information search service to a mobile terminal according to an exemplary embodiment of the present invention.

3, a user of the portable terminal may photograph a specific structure (e.g., an offline shop (or a shop) such as a restaurant, a movie theater, a book store, a mart, a wholesale store, etc.) (Step 302), and transmits the image information of the obtained limb to the information search server 106 together with the information search request message to request information search (step 304). Here, the image information may include GPS information, which may be the information contained in the photograph, or the location information (GPS coordinates) of the portable terminal provided by the portable terminal with the photograph.

That is, when an information retrieval request message is received together with image information of a photograph from a portable terminal, the GPS-address converter 2041 in the area name extracting block 204 extracts GPS information from the image information of the received photograph, Mapping between the extracted GPS information and a plurality of address information stored in the GPS-address mapping DB 2042 converts the GPS information into address information (step 306).

The address-to-local converter 2043 converts the address information into a local name through the mapping between the address information transmitted from the GPS-address converter 2041 and the regions stored in the address-local mapping DB 2044 308).

For example, assuming that the GPS-address converter 2041 has converted the GPS information to "Fuyang-dong, Gwangjin-gu, Seoul," the address-to-local converter 2043 converts the address " .

Next, the character inference unit 2061 in the character string generation block 206 analyzes the image and checks whether character information indicating a specific name (e.g., shop name, shop name, reciprocal name, etc.) exists in the image Steps 310 and 312).

As a result of checking in step 312, if there is character information, the character inference unit 2061 searches for similarity between the extracted character information and a plurality of characters stored in the character DB 2062 (e.g., To infer the candidate characters (step 314).

In this case, the inferred candidate characters are transmitted to the string decider 2065. In the string decider 2065, the similarity degree between the candidate characters transmitted from the character inferring unit 2061 is compared, (Character recognition) (step 316).

As a result of checking in step 312, if there is no character information in the image of the photograph, the image extractor 2063 calculates the content based similarity between the image and the image information stored in the image DB 2064 Candidate images are extracted (step 318).

Again, the string decider 2065 compares the similarity magnitudes of the candidate images transmitted from the image extractor 2063 to determine a candidate image having the relatively largest similarity among the candidate images as a mapped image, The character is determined (recognized) as a character string (image-based character string) (step 320).

Next, the query term generation block 208 automatically generates a query word by combining the local name transmitted from the address-to-local converter 2043 with a character string (character-based string or image-based character string) transmitted from the string decider 2065 To the information search execution block 210 (step 322).

In response, the information search execution block 210 executes a search based on the query term transmitted from the query term generation block 208 to search for a specific structure (e.g., a restaurant, a movie theater, a bookstore, a mart, a wholesale store, (E.g., at least one of a product type and a price, a service type and price, a discount status and a discount, a discount card type, and a description) of the offline shop (or store) (step 324) The acquired related search information is transmitted to the portable terminal through the interface block 202 (step 326). Here, the information retrieval based on the query term can be executed through retrieval of the external web or through retrieval into a retrieval system specialized for a specific structure.

On the other hand, in the present embodiment, the region name is extracted first from the image information of the photograph, and then the character string (character based string or image based character string) is recognized and described. However, The present invention is not necessarily limited to this, and it goes without saying that the process of extracting the area name and the processor recognizing the string can be performed at the same time.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the present invention. It will be readily apparent that such substitutions, modifications, and alterations are possible.

102: Portable terminal group 102/1 - 102 / n: Portable terminal
104: network 106: information retrieval server
202: interface block 204: area name extraction block
206: string generating block 208: query word generating block
210: information search execution block 2041: GPS-address converter
2042: GPS-Address Mapping DB 2043: Address-to-Local Converter
2044: address-area mapping DB 2061: character inference unit
2062: character DB 2063: image extractor
2064: image DB 2065: string determiner

Claims (18)

A step of performing mapping between GPS information and an address and an area of a photographing place of the photograph when an image of a photograph of a specific structure is received from a portable terminal to extract a local name;
Recognizing a character string representing a specific name from the image;
Generating a query word by combining the extracted local name and the recognized character string;
Acquiring related search information of the specific structure through a search based on the created query word, and transmitting the acquired related search information to the mobile terminal
Based information retrieval method.
The method according to claim 1,
The GPS information includes:
The information included in the photograph
Image based information retrieval method.
The method according to claim 1,
The GPS information includes:
The portable terminal providing location information of the portable terminal provided with the photograph
Image based information retrieval method.
The method according to claim 1,
The step of extracting the area name includes:
Converting the GPS information into address information;
Converting the address information into the area name through mapping between the converted address information and the area
Based information retrieval method.
The method according to claim 1,
The step of recognizing the character string may include:
Analyzing the image and extracting character information indicating the specific name;
Inferring candidate characters by searching for similarity between the extracted character information and a plurality of characters stored in the character DB;
Recognizing the candidate character having the relatively largest similarity among the candidate characters inferred as the character string
Based information retrieval method.
The method according to claim 1,
The step of recognizing the character string may include:
Extracting candidate images by calculating a content-based similarity between the image and a plurality of image information stored in the image DB;
Determining a candidate image having a relatively largest degree of similarity among the extracted candidate images as a mapped image;
Recognizing the character corresponding to the determined mapping image as the character string
Based information retrieval method.
The method according to claim 1,
The step of recognizing the character string may include:
Analyzing the image and extracting character information indicating the specific name;
Inferring candidate characters by searching for similarity between the extracted character information and a plurality of characters stored in the character DB;
Recognizing a candidate character having a relatively largest similarity among the candidate characters estimated to be at least larger than a predetermined reference similarity value as the character string;
A step of extracting candidate images through calculation of the content-based similarity between the image and a plurality of image information stored in the image DB when there is no candidate character having a degree of similarity at least greater than the preset reference similarity value among the inferred candidate characters and,
Determining a candidate image having a relatively largest degree of similarity among the extracted candidate images as a mapped image;
Recognizing the character corresponding to the determined mapping image as the character string
Based information retrieval method.
The method according to claim 1,
The search may include:
Runs through search of external web
Image based information retrieval method.
The method according to claim 1,
The search may include:
Is executed through a search into a search system specialized for the specific structure
Image based information retrieval method.
10. The method according to claim 8 or 9,
The related search information may include,
Including at least one of the type and price of the product, the type and price of the service, the discount and the details of the discount, the type of the discount card,
Image based information retrieval method.
An interface block for receiving an image of a photograph of a specific structure transmitted from the mobile terminal,
A region name extracting block for performing mapping between the GPS information, the address and the region of the photographing place of the received photograph to extract a local name,
A character string generating block for recognizing a character string representing a specific name from the image,
A query term generation block for generating a query term by combining the extracted local name and the recognized character string,
Acquiring related search information of the specific structure through a search based on the created query, and providing the acquired related search information to the mobile terminal,
Based information retrieval apparatus.
12. The method of claim 11,
The area name extracting block,
A GPS-address mapping DB storing a plurality of pieces of GPS information and corresponding address information,
A GPS-address converter for extracting the GPS information included in the image and converting the GPS information into address information through mapping between the extracted GPS information and address information stored in a GPS-address mapping DB;
An address-area mapping DB in which a plurality of address information and a plurality of corresponding areas are stored,
And converting the address information into the local name through the mapping between the converted address information and the regions stored in the address-mapping DB,
Based information retrieval apparatus.
13. The method of claim 12,
The GPS information includes:
The information included in the photograph
An image-based information retrieval apparatus.
13. The method of claim 12,
The GPS information includes:
The portable terminal providing location information of the portable terminal provided with the photograph
An image-based information retrieval apparatus.
12. The method of claim 11,
Wherein the character string generation block comprises:
A character DB storing a plurality of characters,
A character inference unit for extracting character information representing the specific name by analyzing the image and deducing candidate characters through the similarity search between the extracted character information and a plurality of characters stored in the character DB;
Determining a candidate character having the largest relative similarity among the candidate characters inferred as the character string,
Based information retrieval apparatus.
12. The method of claim 11,
Wherein the character string generation block comprises:
An image DB storing a plurality of pieces of image information,
An image extractor for extracting candidate images through calculation of content-based similarity between the image and a plurality of pieces of image information stored in the image DB;
Determining a candidate image having a relatively largest degree of similarity among the extracted candidate images as a mapping image and determining a character corresponding to the determined mapping image as the character string,
Based information retrieval apparatus.
12. The method of claim 11,
The information search execution block includes:
Acquiring the related search information through a search on the external web
An image-based information retrieval apparatus.
12. The method of claim 11,
The information search execution block includes:
Acquiring the related search information through a search into a search system specialized for the specific structure
An image-based information retrieval apparatus.

Images