KR20210094396A - Application for searching service based on image and searching server therefor - Google Patents

Abstract

The present invention relates to an image-based search application and to a search server therefor. In particular, provided are the image-based search application for searching for information by generating a query using a photographed image when a search target is photographed in a mobile communication terminal equipped with a camera, and the search server therefor.

Description

APPLICATION FOR SEARCHING SERVICE BASED ON IMAGE AND SEARCHING SERVER THEREFOR

The present invention relates to an image-based search application and a search server therefor, and more particularly, an image-based search application that searches for information by generating a query from the photographed image when a search target is photographed in a mobile communication terminal equipped with a camera. and a search server therefor.

Recently, with the development of IT technology, a method of querying and searching by accessing various search engines or social networks through the Internet is used to obtain information on objects or places.

However, the above method of accessing information has a problem in that it is possible to search based on basic information about the object or place, for example, the product name or region name of the object.

Moreover, even if the basic information about the search target is known, the search requires a lot of time and effort, and even professionalism is required to obtain the desired information in a customized way.

To solve this problem, some items or e-contents are provided with barcodes or QR codes to facilitate information access.

Republic of Korea Patent Publication No. 10-2014-0132028 Republic of Korea Patent Publication No. 10-2018-0058461

The present invention is to solve the above problems, and when a search target is photographed in a mobile communication terminal equipped with a camera, an image-based search application for searching information by generating a query word from the photographed image, and a search server therefor would like to provide

To this end, the image-based search application according to the present invention is an image-based search application driven in a mobile communication terminal, wherein the image-based search application is stored in the memory of the mobile communication terminal and executed by a processor, and the mobile communication a photographing step of photographing a search target with a camera provided in the terminal; a search request step of requesting a search from a search server based on the video image obtained through the photographing; a search result receiving step of receiving a search result based on the video image from the search server; and a search result providing step of classifying the received search results by category and displaying them on the display of the mobile communication terminal.

In this case, the search request step generates a query in a mobile communication terminal by analyzing the captured video image, and then transmits it to a search server or generates a query in a search server that receives the captured video image. It is preferable to include

Preferably, the search request step includes a search condition setting step of setting a search condition in a condition setting unit provided through a user interface of the mobile communication terminal.

In addition, the search condition setting step may include: a first level setting step of requesting a search with the entire video image obtained by photographing the entire search target; and a second level setting step of requesting an additional search for a partial video image corresponding to a portion of the entire video image when the user does not obtain a desired search result by the first level setting step; .

Preferably, the second level setting step is a step of requesting a search for at least any one or more of parts, characters, and symbols included as a part of the entire video image.

In addition, when the user does not obtain a desired search result by the second level setting step, it is preferable to perform a third level setting step of requesting a search with a texture image that appears when the search target is enlarged.

In addition, the search condition setting step preferably includes an identification code setting step of requesting a search by analyzing the identification code when the search target includes an identification code including a barcode and a QR code.

In addition, the search condition setting step may include a location information setting step of obtaining location information of the area where the photographing was made through a GPS module mounted on the mobile communication terminal and requesting a search including the location information. do.

In addition, the search condition setting step may include: a first category providing a description of the search target; a second category providing a social network or community in which the search target is mentioned; a third category providing a homepage for distributing the search target; and a fourth category that provides a site where the transaction of the search target is made; and a category setting step of including at least one category among a plurality of categories in a search condition.

In addition, the search result providing step transmits the search results corresponding to the category selected in the category setting step to the mobile communication terminal, and when a plurality of categories are ordered in order, they are displayed on the display of the mobile communication terminal according to the predetermined order. It is preferable to do

Meanwhile, the image-based search server according to the present invention performs an image search requested by the image-based search application as described above, and includes: a network interface for data communication with the mobile communication terminal; a search information DB for receiving and storing images captured by the mobile communication terminal; and a search engine that searches for the search target by using the query.

In this case, it is preferable to further include an application DB that stores the image-based search application and provides it upon a download request from the mobile communication terminal.

As described above, in the present invention, when a search target is photographed in a mobile communication terminal equipped with a camera, a query word is generated from the photographed image, and related information related to the search target is automatically searched.

Accordingly, when searching for information, even if there is no separate basic information on a search target, if an image is presented by photographing the search target with a camera, the search is automatically enabled based on visual information extracted from the photographed image.

1 is an embodiment showing a system diagram of a service system to which the present invention is applied.
2 is a block diagram showing a mobile communication terminal equipped with an image-based search application according to the present invention.
3 is a flowchart illustrating an execution sequence of an image-based search application according to the present invention.
4 is a flowchart illustrating a search condition setting state in FIG. 3 .
5 is a classification diagram illustrating search conditions for each category in FIG. 3 .
6 is a block diagram illustrating an image-based search server according to the present invention.
7 to 14 are exemplary embodiments showing image-based search results according to the present invention.

Hereinafter, an image-based search application and a search server therefor according to a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

First, as shown in FIG. 1, the service system to which the present invention is applicable includes a mobile communication terminal 10 on the client side and an image-based search server 20 on the server side, and they perform two-way data communication through a wired/wireless communication network.

At this time, the mobile communication terminal 10 requests the search server 20 to search based on a photographed image, that is, time to receive a search result, and is equipped with a camera such as a smartphone, tablet PC, PDA, smart pad, and netbook. It includes a variety of communication terminals.

The image-based search server 20 (hereinafter referred to as a 'search server') searches for related information using a query generated based on the image captured by the mobile communication terminal 10 as described above, and the result is provided to the mobile communication terminal (10).

In this respect, the search server 20 of the present invention corresponds to the time-based search server 20, and the search server 20 uses its own built-in search engine or an external third server ( An external search engine) 30 is used to search for information.

As described above, in the present invention, when a search target is photographed in the mobile communication terminal 10 equipped with a camera, a query word is generated from the photographed image, and information related to the search target and/or information related thereto is automatically searched. .

Accordingly, even if there is no separate basic information on the search target when searching for information, as long as an image is provided by photographing the search target with the camera, the search is automatically enabled based on the visual information extracted from the photographed image.

To this end, the image-based search application according to the present invention is implemented in the mobile communication terminal 10 as described above, stored in the memory 15 of the mobile communication terminal 10 and executed by the processor 16 .

As shown in FIG. 2 , the mobile communication terminal 10 has a camera 11 , a display 12 , a GPS module 13 , a communication module 14 , a memory 15 and a processor for the unique functions of the terminal. Including (16), these hardware configurations are linked with the technical configurations of the present invention as follows.

According to the technical configurations of the present invention, the UI processing unit 17, the search request unit 18 and the condition setting unit 19 are connected to the mobile communication terminal 10 by executing the application stored or downloaded in the mobile communication terminal 10. is implemented

In this case, the UI processing unit 17 provides various interfaces (UI: User Interface) to the user when the application is executed in the mobile communication terminal 10 . Such an interface includes a search request unit 18 and a condition setting unit 19 displayed on the terminal, and is implemented as, for example, a GUI-type selection button and a text input box.

Accordingly, the search request unit 18 is provided on the screen (display) of the mobile communication terminal 10 by the UI processing unit 17 and is used to request a search from the search server 20 as described below. The condition setting unit 19 is used to set search conditions such as a search range.

3 shows a search service sequence implemented in the mobile communication terminal 10 by the image-based search application according to the present invention.

As shown, the image-based search application according to the present invention is a photographing step (S10), a search request step (S20), and a search result receiving step (S30) so that the mobile communication terminal 10 can search for information based on time. and a search result providing step (S40).

In this case, the photographing step (S10) is a step of photographing a search target using the camera 11 mounted on the mobile communication terminal 10, and the search target includes a person (person), an object (article), and a structure (building). and places (foreground), and so on.

As such, the video image obtained through photographing is directly used for information retrieval. Accordingly, the present invention makes it possible to search for information based on a video image even if the basic information used in the conventional information search, such as a product name or an area name, is not known.

Next, in the search request step (S20), a search is requested to the search server 20 based on the video image obtained through the shooting in the mobile communication terminal 10, and the search server 20 accesses the network if there is a search request. Then, a search (S21) is performed using the query word.

Here, the query becomes the video image itself and contrasts between images, or becomes a character string inferred or extracted from the video image by image information processing and machine learning (eg, AI-based deep learning, etc.), or It can be a combination of these images and strings.

In addition, the query word may be generated by the mobile communication terminal 10 itself or by the search server 20 receiving a search request. When a query word is generated by the mobile communication terminal 10 itself by analyzing the captured video image, the generated query word is transmitted to the search server 20 to be used for search.

However, in consideration of the data throughput and speed of the mobile communication terminal 10 such as a smart phone, it is preferable that the query word is generated by the search server 20 . Furthermore, it is desirable to set a condition when searching, which will be described again below.

Next, in the search result receiving step ( S30 ), a search result based on the video image is received from the search server 20 . As described above, the search server 20 searches only with its own search engine, or searches using a third server 30 built outside if necessary.

The search result is received through the communication module 14 of the mobile communication terminal 10 being connected to the search server 20 through a wired/wireless communication network. It is preferable to provide a plurality of search results classified by category.

In the search result providing step (S40), the received search results are classified by category (refer to FIG. 5) and displayed on the display 12 of the mobile communication terminal 10, and if necessary, the search results including hyperlinks are provided.

The categories can be variously divided into 'information' for explaining the search target, a 'social network' commonly referred to as SNS, a 'homepage' for managing the search target, and a 'transaction site' for trading.

In addition, by using the category classified as above as the upper category, sub-categories can be re-classified for each upper category, and the user can select a desired category to search.

In addition, the user provided with the search result can determine whether the search result satisfies his or her desired requirements and decide to terminate the search (S41). can do.

Meanwhile, as described above, when a search is requested from the search server 20 in the search request step, the present invention may further proceed with the search condition setting step S20a. The set search condition increases search accuracy and search probability in image-based search.

To this end, in the search condition setting step (S20a), a condition setting unit 19, which is a user interface (UI), is provided on the screen of the mobile communication terminal 10, and a search condition is input from the user through the provided condition setting unit 19. receive

The condition setting unit 19 is an input window or selection button displayed on the display 12 of the mobile communication terminal 10 under the control of the UI processing unit 17, and is an input window to set each condition described below. and/or a selection button is created and provided.

4 shows the setting conditions used in the image-based search in the present invention step by step.

As shown, the present invention is a first level setting step (CD-1), a second level setting step (CD-2), a third level setting step (CD-3), an identification code setting step (CD-4) , a location information setting step (CD-5) and a category setting step (CD-6), each of which is activated by pressing a selection button displayed on the mobile communication terminal 10 and receives a condition input.

Here, the first to third level setting steps CD-3 is a condition setting step according to the adjustment of the range of the video image, and the identification code setting step CD-4 is a condition setting step using the identification code when it exists.

In addition, the location information setting step CD-5 is a condition setting step using GPS location information, and the category setting step CD-6 is a condition setting step for selecting a category divided according to a search result.

More specifically, the first level setting step (CD-1) is a step of requesting a search with the entire video image obtained by photographing the entire search target. If there is no separate condition setting, the first level setting step (CD-1) ) is the default, and the entire video image is searched.

The second level setting step CD-2 is used when the user does not obtain a desired search result by the first level setting step CD-1 or when the search fails, and a portion corresponding to a portion of the entire video image Request additional search by video image.

In the case of a search using the entire video image, since there are many features or objects in one image, the search failure or noise may occur because the feature or object cannot be extracted during the search and is judged as a whole.

Accordingly, the user, for example, enters the second level setting step (CD-2) by pressing a GUI button provided on the screen, and then requests a search by designating a certain area or re-photographing only a certain part of the called whole image.

A certain area of the entire image corresponds to the additional search area. As a preferred embodiment, a part of the entire video image includes parts, characters, and symbols included in the search target, and a re-search is requested using this.

For example, if the search target is a car (all), if a satisfactory search is not obtained only in the first level setting step CD-1, the search is further limited to headlights (parts) in the second level setting step CD-2. make this happen

Similarly, if the search target is a building (all) found in a travel destination, if a satisfactory search is not obtained only in the first level setting step (CD-1), in the second level setting step (CD-2), the signboard ( character) to further limit the search.

In addition, the symbol may include an emblem attached to a vehicle, a building sign, or a trademark attached to an article, and through this second level setting step (CD-2), the first level setting step (CD-1) ) can be searched more precisely.

Finally, in the third level setting step CD-3, when the user does not obtain a desired search result by the second level setting step CD-2, the search target is searched with a texture image that appears when the target is enlarged. This is a request step.

Even in the third level setting stage (CD-3), the third level setting stage (CD-3) is entered by pressing the GUI button provided on the screen, and the user resets the resolution for the entire called image or zooms in (zoom-) In) state, retake a photo and request a search.

In this way, in the case of adding texture as a search requirement in the third level setting step (CD-3), it is mainly effective for textured articles (eg, fibers or wood). In the case of a search target, the search efficiency is improved.

In the identification code setting step (CD-4), which has not been described above, if the search target includes an identification code including a barcode and a QR code, the identification code is analyzed and a search is requested.

That is, in the identification code setting step (CD-4), after checking whether an identification code is included in the photographed search target, if there is an identification code, it is searched using the identification code. Of course, even when searching using the identification code, the condition setting of the first to third level setting steps CD-3 described above may also be used, and this is also the case hereafter.

In the location information setting step (CD-5), the location information of the area where the photographing was taken is obtained through the GPS module 13 mounted on the mobile communication terminal 10, and a query word including the location information of the area is generated and searched. to request

The location information can be directly provided from the GPS module 13 of the mobile communication terminal 10 or can be used as location information by extracting what is included as metadata in a photographed video image file, and more easily searching through location information. do.

However, in the case of adding location information as a search condition, it is mainly effective when the search target is only in a specific area, such as a specific object or famous scenery, so it is preferable to use the location information setting step (CD-5) only in this case.

Next, in the category setting step (CD-6), the search time is shortened by limiting (restricting) the search to items corresponding to the set category when searching for information, and is classified into four categories, for example.

As shown in FIG. 5 , the four categories are classified into four categories of the first to fourth categories, and each of the first to fourth categories can be reclassified into a lower category.

In this case, the first category provides a description of the search target and is mainly related to information provision such as public institutions, education, and literature, and the second category is a social network or community in which the search target is mentioned (posted, ripple). is related to

The third category is related to the homepage that distributes the search target, and it corresponds to an Internet site hosted by a company or event. The fourth category provides a site where a search target transaction is made, and includes an open market or a used site.

Accordingly, the user prevents a search other than the field of interest from being performed and significantly increases the search speed by including at least one category among a plurality of categories in the search condition.

Furthermore, if the above condition setting is made, the limitation is also made in the search result providing step (S40) in synchronization with it. That is, only the search results corresponding to the category selected in the category setting step CD-6 are transmitted to the mobile communication terminal 10 and provided.

In this case, a plurality of categories are included as a search condition, but when these categories are ordered, they are displayed on the display 12 of the mobile communication terminal 10 according to the determined order, thereby providing a user-friendly environment.

Meanwhile, the image-based search server 20 according to the present invention performs an image search requested by an image-based search application, and transmits the search result to the client-side mobile communication terminal 10 .

To this end, the search server 20 includes a network interface 21 , a search information DB 22 , a query word generator 23 , and a search engine 24 . In addition, as a basic configuration constituting the server, it includes a server memory 25 and a central processing unit 26 .

Here, the network interface 21 performs data communication with the mobile communication terminal 10 and receives a captured video image and a search request command from the mobile communication terminal 10 . In addition, in some cases, a query word generated by the mobile communication terminal 10 is also input.

The search information DB 22 receives and stores an image captured by the mobile communication terminal 10 , and the stored video image is used to generate a query word. Even when a query word is generated on the server side, the query word is stored in the DB (DataBase) together with the client ID.

The query word generator 23 is used to generate a query word using the video image captured by the search server 20. In this way, when the search server 20 generates a query word, the Only files are provided.

The search engine 24 searches for information related to a search target using the query word generated as described above, and the search engine 24 is installed in the search server 20 or some of it is located in the external server 30 . It may be built and linked together.

However, it is preferable that the search server 20 further includes an application DB 27 , and the application DB 27 stores an image-based search application so that the mobile communication terminal 10 can provide it upon a download request. Accordingly, the search server 20 of the present invention can also function as an application download server.

Hereinafter, various embodiments of an image-based search application and a search server therefor according to the present invention as described above will be described.

7 shows a state in which a search request is made to the search server 20 by photographing an advertisement attachment plate with the mobile communication terminal 10 possessed by the user. is provided

At this time, since a plurality of advertisements are attached to the advertisement attachment board, more accurate information is provided by setting a search condition and designating a specific advertisement. For example, through the second level setting step CD-2, a portion having a specific advertisement is reselected from among the entire video image to request a search, and only the search result is provided.

FIG. 8 is a photograph of the back side of a US dollar bill with the mobile communication terminal 10. After shooting, when a search request is made to the search server 20, a site indicating the current value of one dollar is searched. Of course, it could also be linked to the meaning of the picture on the one dollar bill or the Mint in the United States.

9 is a photograph of the front of a restaurant, and when a search request is made after shooting, the search server 20 provides a menu board and a community space of the corresponding restaurant. In this case, it is possible to set conditions to search for signboard text included in a part of the building rather than the image of the entire building.

10 illustrates a state in which a search is requested by photographing the Eiffel Tower in Paris, France during a trip. Attractions such as the Eiffel Tower are very famous and you know the name, but you can find various categories such as the history of the Eiffel Tower, its height, and how to take good pictures through image search through photography. In this case, if the GPS location information at the time of shooting is added as a setting condition, the search accuracy can be further increased.

11 shows a request for a search after photographing the exercise equipment, and introduces a site that informs an effective exercise method as a result of the search. Of course, the city is omitted, but the name of the exercise equipment, the sales site, and the nearest gym (gym) from the shooting location may be searched and introduced.

In addition, if a search is requested after photographing a plant as shown in FIG. 12, a site providing detailed information about the plant is searched and provided, and if a person is photographed as shown in FIG. to provide In addition, when food as shown in FIG. 14 is photographed, a recipe for it is searched for and provided.

Therefore, in the present invention, even when basic information such as a product name or an area name for a search target is not known, by simply taking a picture with the mobile communication terminal 10 possessed by the user and requesting a search, various categories of search information can be provided. be able to

In the above, specific embodiments of the present invention have been described above. However, the spirit and scope of the present invention is not limited to these specific embodiments, but various modifications and variations are possible within the scope that does not change the gist of the present invention. You will understand when you grow up.

Therefore, since the embodiments described above are provided to fully inform those of ordinary skill in the art to which the present invention pertains the scope of the invention, it should be understood that they are exemplary in all respects and not limiting, The invention is only defined by the scope of the claims.

10: mobile communication terminal 11: camera
12: display 13: GPS module
14: communication module 15: memory
16: terminal central processing unit 17: UI processing unit
18: search request unit 19: condition setting unit
20: search server 21: network interface
22: search information DB 23: query word generator
24: search engine 25: server memory
26: server central processing unit 27: application DB

Claims (12)

In the image-based search application driven in the mobile communication terminal 10,
The image-based search application is stored in the memory 15 of the mobile communication terminal 10 and executed by the processor 16,
a photographing step (S10) of photographing a search target with a camera 11 provided in the mobile communication terminal 10;
a search request step (S20) of requesting a search from the search server 20 based on the video image obtained through the photographing;
a search result receiving step of receiving a search result based on the video image from the search server 20 (S30); and
An image-based search application, characterized in that it is implemented, including; a search result providing step (S40) of classifying the received search results by category and displaying them on the display 12 of the mobile communication terminal (10). According to claim 1,
The search request step (S20),
The mobile communication terminal 10 generates a query by analyzing the captured video image, and then transmits it to the search server 20 or generates a query in the search server 20 that receives the captured video image An image-based search application comprising the steps. According to claim 1,
The search request step (S20),
and a search condition setting step (S20a) of setting a search condition in the condition setting unit (19) provided through the user interface of the mobile communication terminal (10). 4. The method of claim 3,
The search condition setting step (S20a) is,
a first level setting step (CD-1) of requesting a search with the entire video image of the entire search target; and
When the user does not obtain a desired search result by the first level setting step CD-1, a second level setting step of requesting an additional search with a partial video image corresponding to a part of the entire video image (CD-) 2); Image-based search application comprising a. 5. The method of claim 4,
The second level setting step (CD-2) is,
The image-based search application, characterized in that the step of requesting a search with at least any one or more of parts, characters, and symbols included as a part of the entire video image. 5. The method of claim 4,
If the user does not obtain the desired search result by the second level setting step CD-2,
The image-based search application further comprising a third level setting step (CD-3) of requesting a search with a texture image that appears when the search target is enlarged. 4. The method of claim 3,
The search condition setting step (S20a) is,
and an identification code setting step (CD-4) of analyzing the identification code and requesting a search when the search target includes an identification code including a barcode and a QR code. 4. The method of claim 3,
The search condition setting step (S20a) is,
A location information setting step (CD-5) of obtaining location information of the area where the photographing was made through the GPS module 13 mounted on the mobile communication terminal 10 and requesting a search including the location information is included. An image-based search application, characterized in that. 4. The method of claim 3,
The search condition setting step (S20a) is,
a first category providing a description of the search target;
a second category providing a social network or community in which the search target is mentioned;
a third category providing a homepage for distributing the search target; and
image-based search comprising a category setting step (CD-6) of including at least any one or more categories among a plurality of categories including a fourth category providing a site where the transaction of the search target is made in a search condition; application. 10. The method of claim 9,
The search result providing step (S40) is,
A search result corresponding to the category selected in the category setting step (CD-6) is transmitted to the mobile communication terminal 10,
When the order of a plurality of categories is determined, the image-based search application, characterized in that it is displayed on the display 12 of the mobile communication terminal 10 according to the predetermined order. In the imager-based search server for performing an image search requested by the image-based search application as in any one of claims 1 to 10,
a network interface 21 for data communication with the mobile communication terminal 10;
a search information DB 22 for receiving and storing an image photographed by the mobile communication terminal 10; and
and a search engine (24) that searches for the search target by using the query. 12. The method of claim 11,
and an application DB (27) that stores the image-based search application and provides it upon a download request from the mobile communication terminal (10).

Images