KR20110017171A - System, server, mobile terminal and method for providing of image searching service using search query - Google Patents

Abstract

PURPOSE: A service providing system of image searching service, a server, a mobile terminal and a method thereof using a search query are provided to search an image data which a user desire, thereby increasing an accuracy of image data search. CONSTITUTION: A search server(30) extracts a candidate image data by searching a search query from a mobile terminal by receiving the search query with a text data. The search server(40) transmits an image data including the candidate image data and the search query. The image processing server transmits the similarity to search server by extracting similarity of each image data from a feature vector of an image data comprising the candidate data and a search query.

Description

System, Server, Mobile Terminal and Method for Providing of Image Searching Service Using Search Query}

The present invention relates to an image data retrieval technology, and more particularly, to an image data retrieval technique of a candidate image data extracted from a search performed as text data among image data and text data included in a search query and image data included in a search query. The present invention relates to an image search service providing system, a server, a mobile terminal, and a method using a search query for providing a mobile terminal with search results generated based on a similarity calculated by comparing feature vectors.

Recently, with the development of multimedia technology, the usage of multimedia data has increased in each field. As a conventional multimedia data retrieval method, a text retrieval method is typical.

The text search method is a method of performing a text-based search by indexing features of multimedia data into text and inputting a desired keyword. This text retrieval method may not only extract features inefficiently, but also may cause a problem that a large amount of false multimedia retrieval results are obtained due to limitations of text.

In particular, in the case of a wireless Internet search using a mobile terminal, input of a search query is limited due to the limited screen size and inconvenience of keypad input of the mobile terminal. In addition, when the user wants to search for the desired multimedia data, if only the text keyword is used, the multimedia data related to the text keyword is searched too much, which causes a limitation in the user searching the search results and incurs a large search cost. There is this.

In order to solve the conventional problems, an object of the present invention is to provide a video search service providing system, a server, a mobile terminal and a search query using the search query to search for the desired video data using a search query including text data and video data and To provide a way.

In order to achieve this object, an image search service providing system using a search query according to the present invention performs a search with the text data when the search query is received from a mobile terminal generating a search query composed of text data and image data. A search server for transmitting the extracted candidate image data and the image data included in the search query to an image processing server, generating a search list based on the similarity received from the image processing server, and transmitting the searched image data to the mobile terminal; And the image processing server extracting a similarity degree of each image data from the feature vector of the candidate image data received from the server and the image data constituting the search query and transmitting the similarity of the image data to the search server.

In addition, the image search service providing search server using a search query according to the present invention is a mobile terminal, a communication unit for communicating with the image processing server, when receiving a search query consisting of text data and image data from the mobile terminal to the text data; After performing the search, the extracted candidate image data and the image data included in the search query are transmitted to an English wound server, a search list is generated based on the similarity received from the image processing server, and then transmitted to the mobile terminal. It characterized in that it comprises a control unit.

The control unit may receive the similarity calculated from the feature vector comparison result of the candidate image data and the image data from the image processing server.

The controller may set the priority of the candidate image data based on the received similarity.

The control unit may generate a search list by sorting the candidate image data according to the set priority, and transmit the generated search list to the mobile terminal.

The control unit may generate the search list in the form of a thumbnail, and when the selection signal for the at least one image data of the search list is received from the portable terminal, extract the image data corresponding to the selection signal to the portable terminal. It characterized in that the transmission to.

The apparatus may further include a storage unit which stores a plurality of image data which can be searched using the search query.

In addition, the image processing server providing a search service using a search query according to the present invention is a communication unit for communicating with a search server performing a search using a search query received from a mobile terminal, the candidate image data and the search from the search server And a control unit for receiving the image data included in the query, comparing the feature vectors of the respective image data, calculating the similarity degree of each image data, and transmitting the image data to the search server.

The controller may extract a feature vector of each of the image data.

The feature vector may be a vector extracted from the feature points by extracting feature points of a subject present in each of the image data.

In addition, the mobile terminal providing a search service using a search query according to the present invention enters a communication unit performing communication with a search server, an image search menu, and transmits a search query generated from text data and image data to the search server. Outputs a search list for the search query received from a search server, and if a selection signal for at least one of the search lists is input, transmits the selection signal to the search server and then sends the search signal to the selection signal received from the search server. And a controller for outputting corresponding image data.

In addition, in a method of providing a search service using a search query according to the present invention, a search server receives a search query including text data and image data from a mobile terminal, and the search server receives the text data included in the received search query. Performing a search by the search server, extracting candidate image data which is the search result, and transmitting the candidate image data and the image data included in the search query to an image processing server; Generating a search list based on the similarity received from the image processing server, and transmitting the generated search list to the mobile terminal by the search server.

In addition, the method for providing a search service using a search query according to the present invention receives candidate image data and image data included in the search query from a search server where the image processing server performs a search using a search query received from a mobile terminal. And comparing, by the image processing server, a feature vector of the image data included in the search query with the candidate image data, and by the image processing server, calculating a similarity degree of each image data according to the comparison result. And transmitting to the server.

In addition, the method for providing a search service using a search query according to the present invention includes the steps of a mobile terminal entering an image search menu, the mobile terminal transmitting a search query generated from text data and image data to a search server, and the mobile terminal. Outputting a search list for the search query received from the search server by the terminal; transmitting the selection signal to the search server when a selection signal for at least one of the search lists is input by the portable terminal; And a mobile terminal receiving and outputting image data corresponding to the selection signal from the search server.

As described above, the present invention can improve the accuracy of image data search by retrieving image data desired by a user using a search query including text data and image data, and enhance the convenience and usability of the search service. There is.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, in describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present invention belongs and are not directly related to the present invention are omitted. This is to more clearly communicate without obscure the core of the present invention by omitting unnecessary description.

Hereinafter, in an embodiment of the present invention, the mobile terminal may be a mobile communication terminal, a personal digital assistant (PDA), a smart phone, a personal computer, a notebook, or the like for providing convenience to a user. have.

1 is a block diagram showing the main configuration of a search service providing system using a search query according to an embodiment of the present invention. 2 is a block diagram showing the main configuration of the mobile terminal shown in FIG.

1 and 2, the search service providing system 100 according to the present invention includes a mobile terminal 10, a communication network 20, a search server 30, and an image processing server 40.

The mobile terminal 10 generates a search query according to a user's input and transmits the search query to the search server 30, outputs a search list for the search result received from the search server 30, and the image data included in the search list. Outputs

To this end, the mobile terminal 10 includes a first communication unit 11, an input unit 12, a camera 13, an output unit 14, a first storage unit 15, and a first control unit 16.

The first communication unit 11 performs communication with the search server 30 through the communication network 20.

The input unit 12 transmits a signal corresponding to a key input from a user to the first control unit 16 to control the operation of the mobile terminal 10. The input unit 12 may be configured as a conventional keypad, and may be configured as a touch screen, a touch pad, a scroll wheel, or the like.

The camera 13 is activated under the control of the first control unit 16 to obtain image data of the subject. The camera 13 includes an image sensor for converting an optical signal of a subject into an analog signal and a signal processor (not shown) for converting an analog signal into a digital signal. That is, the camera 13 generates image data by converting the optical signal collected through the lens into a digital signal and provides it to the first controller 16.

The output unit 14 displays a series of operation states, operation results, and a plurality of pieces of information made in the mobile terminal 10 under the control of the first control unit 16. In this case, the output unit 14 may include a display device such as a liquid crystal display (LCD), organic light emitting diodes (OLED), a speaker, a vibration sensor capable of generating vibrations in the mobile terminal 10, and the like. have. In addition, when the output unit 14 is configured as a touch screen, it can simultaneously perform the functions of the input unit 12 and the display device.

The first storage unit 15 stores a program or the like for controlling the mobile terminal 10 under the control of the first control unit 16.

In particular, the image DB 15a of the first storage unit 15 stores the image data obtained from the camera 13 through the first control unit 16, and the query DB 15b of the first control unit 16 The search query generated from the text data input from the input unit 12 and the image data obtained from the camera 13 is stored. In addition, the first storage unit 15 stores a program for generating a search query with text data and image data.

The first control unit 16 is in charge of controlling the overall operation of the mobile terminal 10.

In particular, the first controller 16 enters the image search menu by the input of the input unit 12. The first controller 16 receives text data, which is a keyword for retrieving image data, from the input unit 12, and receives image data. In this case, the image data is selected by acquiring image data from the camera 13 after activating the camera 13 by the input of the input unit 12, or at least one image data previously stored in the first storage unit 15. Choose from.

The first controller 16 drives a program stored in the first storage unit 15 to generate a search query using the input text data and the selected image data. The first controller 16 transmits the generated search query to the search server 30 and controls the output unit 14 to output the search list received from the search server 30.

The first control unit 16 transmits the selection signal to the search server 30 when the selection signal for the specific image data is input from the input unit 12 from the search list, and receives the selection signal received from the search server 30. The output unit 14 is controlled to output corresponding image data.

The communication network 20 generates a communication path between the mobile terminal 10, the search server 30, and the image processing server 40 to manage communication.

When the search server 30 receives the search query composed of the text data and the image data from the mobile terminal 10, the search server 30 performs the search with the text data to extract the extracted candidate image data and the image data included in the search query. 40, and generates a search list based on the similarity received from the image processing server 40 and transmits the search list to the mobile terminal 10.

To this end, the search server 30 includes a second communication unit 31, a second storage unit 32, and a second control unit 33.

The second communication unit 31 communicates with the mobile terminal 10 and the image processing server 40 through the communication network 20.

The second storage unit 32 stores a program or the like for controlling the search server 30 by the second control unit 33.

In particular, the second storage unit 32 stores a plurality of image data corresponding to the search query received from the mobile terminal 10. The second storage unit 32 stores keywords and image data associated with the plurality of image data, respectively. Map and save.

The second control unit 33 is in charge of controlling the overall operation of the search server 30.

In particular, the second controller 33 searches the image data stored in the second storage unit 32 with the text data included in the search query received from the mobile terminal 10. In this case, the second controller 33 searches whether the same keyword as the text data exists in the second storage unit 32, and if the same keyword as the text data exists in the search result second storage unit 32, the second controller 33 maps to the searched keyword. The stored video data is extracted as candidate video data.

The second controller 33 transmits the extracted candidate image data and the image data included in the search query to the image processing server 40. The second controller 33 transmits the image processing server 40 to the image processing server 40 so as to calculate the similarity between the candidate image data and the image data included in the search query.

The second controller 33 sets the priority of candidate image data extracted based on the similarity received from the image processing server 40, and generates a search list by arranging candidate image data according to the set priority. When generating the search list, the second controller 33 may generate the search list by forming candidate image data in the form of thumbnails.

As such, the search server 30 may perform a search with text data and provide a clearer search result to the user based on the similarity received from the image processing server 40.

The second control unit 33 transmits the generated search list to the mobile terminal 10, and when the selection signal is received from the mobile terminal 10, extracts image data corresponding to the received selection signal to the mobile terminal 10. send.

The image processing server 40 receives the candidate image data and the image data included in the search query from the search server 30 that performs the search by using the search query received from the mobile terminal 10, and then uses the feature vector of each image data. Compare and calculate the similarity of each image data and transmit to the search server (30).

To this end, the image processing server 40 includes a third communication unit 41, a third storage unit 42, and a third control unit 43.

The third communication unit 41 communicates with the search server 30 through the communication network 20.

The third storage unit 42 stores a program or the like for controlling the image processing server 40 by the third control unit 43.

In particular, the third storage unit 42 analyzes the candidate image data received from the search server 30 and the image data included in the search query to extract the feature points of the subject present in each image data, and the extracted feature points. Save the program that extracts the vector. The third storage unit 42 stores a program for calculating a similarity degree of each image data by comparing the feature vectors of the respective image data.

The third control unit 43 is in charge of controlling the overall operation of the image processing server 40.

In particular, the third controller 43 drives a program stored in the third storage 42 to analyze candidate image data received from the search server 30 and image data included in the search query. The third controller 43 extracts a subject present in the analyzed image data and extracts feature points of the extracted subject. The third controller 43 extracts the feature vector using the extracted coordinate values of the feature points.

The third controller 43 drives a program stored in the third storage 42 to compare feature vectors extracted from each image data to calculate similarity for each image data. In particular, the third controller 43 compares the feature vector of the candidate image data with the feature vector of the image data included in the search query and calculates the similarity from the feature vector of the image data included in the search query and the most candidate image data. do.

For example, the third controller 43 receives candidate image data 1, 2, 3 from the search server 30, and compares the candidate image data 1, 2, 3 with a feature vector of the image data included in the search query. As a result, when the candidate image data 2 has 90% similarity with the image data included in the search query, the candidate image data 1 has a similarity of 70% and the candidate image data 3 has a similarity of 65%, the third controller 43 Transmits each candidate image data and the calculated similarity to the search server 30.

The search server 30 receives the generated search list in the order of candidate image data 2, 1, and 3 and transmits the search list to the mobile terminal 10. FIG.

In addition, in the embodiment of the present invention, it is described that the search server 30 and the image processing server 40 exist separately, but are not necessarily limited thereto, and the search server 30 and the image processing server ( All of the functions in 40) can be performed.

3 is a flowchart illustrating a search service providing method using a search query according to an embodiment of the present invention.

Referring to FIG. 3, in operation S51, the mobile terminal 10 enters an image search menu according to an external input.

In operation S53, the mobile terminal 10 receives text data related to the image data to be searched, and in operation S55, the mobile terminal 10 receives image data related to the image data to be searched. In this case, if the image data related to the image data to be searched exists in the first storage unit 15 of the mobile terminal 10, the first storage unit 15 selects the image data, and the image data is stored in the first storage unit 15. If not present, the mobile terminal 10 activates the camera 13 to acquire and select image data.

Thereafter, in step S57, the mobile terminal 10 generates a search query with the input text data and the selected image data.

In operation S59, the mobile terminal 10 transmits the generated search query to the search server 30.

The search server 30 receiving the search query from the mobile terminal 10 performs a search with the text data included in the search query in step S61. In this case, the search server 30 extracts image data related to text data from image data previously stored in the second storage unit 32. Of course, the image data pre-stored in the second storage unit 32 may be mapped and stored with a specific keyword, and the search server 30 may extract the image data mapped to the specific keyword by searching for a specific keyword related to the text data. Can be.

In operation S63, the search server 30 extracts candidate image data from the search result. In operation S65, the search server 30 transmits the candidate image data and the image data included in the search query to the image processing server 40. .

In operation S67, the image processing server 40 extracts the feature vector of the candidate image data received from the search server 30 and the image data included in the search query.

More specifically, the image processing server 40 extracts a subject present in each image data by analyzing candidate image data and image data included in a search query, and extracts a plurality of feature points constituting the extracted subject. The image processing server 40 may extract a feature vector from the connection of the extracted feature points.

In operation S69, the image processing server 40 compares the feature vectors extracted from each image data, and in operation S71, the image processing server 40 calculates the similarity degree of each image data from the comparison result.

In operation S73, the image processing server 40 transmits the calculated similarity to the search server 30. In operation S75, the search server 30 sets priority of candidate image data based on the received similarity. A search list is generated by using thumbnails of candidate image data having priorities set.

For example, the image processing server 40 receives candidate image data 1, 2, 3 from the search server 30, and compares the candidate image data 1, 2, 3 with a feature vector of the image data included in the search query. As a result, when the candidate image data 2 has 90% similarity with the image data included in the search query, the candidate image data 1 has 70% similarity and the candidate image data 3 has 65% similarity, the image processing server 40 Transmits each candidate image data to the search server 30 but also transmits the calculated similarity.

Receiving this, the search server 30 generates a search list in the order of candidate video data 2, 1, and 3, but generates a search list using thumbnails of the candidate video data.

In step S77, the search server 30 transmits the generated search list to the mobile terminal 10, and the mobile terminal 10 receiving the search list in step S79 outputs the search list.

Thereafter, in step S81, the portable terminal 10 waits for input of the selection signal when the selection signal for at least one image data from the search list is not input from the outside, and when the selection signal is input, the portable terminal 10 in step S83. Transmits the input selection signal to the search server 30.

In operation S85, the search server 30 extracts image data corresponding to the selection signal, and in operation S87, the search server 30 transmits the extracted image data to the mobile terminal 10.

Finally, in step S89, the mobile terminal 10 outputs the image data transmitted from the search server 30.

As described above, since the present invention searches for image data using text data and image data, it is possible to provide a search service for image data with improved accuracy.

While the invention has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined by the appended claims.

The present invention relates to an image search service providing system, a server, a mobile terminal, and a method using a search query, wherein the image data is searched by performing a search using image data and input text data pre-stored in the mobile terminal. By improving the accuracy of the user, it is possible to improve the user's satisfaction, thereby increasing the revenue of the mobile terminal manufacturers and service providers.

1 is a block diagram showing the main configuration of a search service providing system using a search query according to an embodiment of the present invention

2 is a block diagram showing the main configuration of the mobile terminal shown in FIG.

3 is a flowchart illustrating a search service providing method using a search query according to an embodiment of the present invention.

<Description of main parts of drawing>

10: mobile terminal 11: first communication unit

12: input unit 13: camera

14: output unit 15: first storage unit

15a: Image DB 15b: Query DB

16: first controller 20: communication network

30: search server 31: the second communication unit

32: second storage unit 33: second control unit

40: image processing server 41: third communication unit

42: third storage unit 43: third control unit

100: search service providing system

Claims (14)

When the search query is received from a mobile terminal that generates a search query consisting of text data and image data, the search is performed on the text data to transmit the extracted candidate image data and the image data included in the search query to the image processing server. A search server generating a search list based on the similarity received from the image processing server and transmitting the search list to the mobile terminal; The image processing server extracting a similarity degree of each image data from the candidate vector data received from the search server and a feature vector of the image data constituting the search query and transmitting the extracted similarity to each search server; Image search service providing system using a search query, characterized in that it comprises a.  A communication unit for communicating with the mobile terminal and the image processing server; Receiving a search query composed of text data and image data from the mobile terminal, the candidate image data extracted by performing a search with the text data and the image data included in the search query are transmitted to an image processing server, and the image processing is performed. A control unit for generating a search list based on the similarity received from the server and transmitting the search list to the mobile terminal; Image search service providing search server using a search query, characterized in that it comprises a. The method of claim 2, The control unit And a similarity calculated as a result of comparing the candidate image data with the feature vector of the image data from the image processing server. The method of claim 3, The control unit Search server providing a search service using a search query, characterized in that for setting the priority of the candidate image data based on the received similarity. The method of claim 4, wherein The control unit Search server providing a search service using a search query, characterized in that to generate a search list by sorting the candidate image data to the set priority and to transmit the generated search list to the mobile terminal. The method of claim 5, The control unit The search list is generated in the form of a thumbnail, and when a selection signal for at least one image data of the search list is received from the mobile terminal, image data corresponding to the selection signal is extracted and transmitted to the mobile terminal. Search service providing search server using a search query. The method of claim 2, A storage unit which stores a plurality of image data which can be searched using the search query; Search service providing search server using a search query, characterized in that it further comprises. A communication unit configured to communicate with a search server performing a search by using a search query received from the mobile terminal; A controller which receives candidate image data from the search server and image data included in the search query, compares feature vectors of each image data, calculates a similarity degree of each image data, and transmits the image data to the search server; Image processing server providing a search service using a search query, characterized in that it comprises a. The method of claim 8, The control unit Search processing providing image processing server using a search query, characterized in that for extracting the feature vector of each image data. 10. The method of claim 9, The feature vector is Search processing providing image processing server using a search query, characterized in that the vector extracted from the feature points by extracting the feature points of the subject present in each of the image data. Communication unit for performing communication with the search server; The user enters the image search menu and transmits a search query generated from text data and image data to the search server, outputs a search list for the search query received from the search server, and selects at least one of the search lists. A control unit for transmitting the selection signal to the search server and outputting image data corresponding to the selection signal received from the search server when a signal is input; Mobile terminal providing a search service using a search query, characterized in that it comprises a. Receiving, by the search server, a search query composed of text data and image data from the mobile terminal; Performing a search by the search server with the text data included in the received search query; Extracting, by the search server, candidate image data that is the search result, and transmitting the candidate image data and the image data included in the search query to an image processing server; Generating, by the search server, a search list based on the similarity received from the image processing server; Transmitting, by the search server, the generated search list to the mobile terminal; Search service providing method using a search query, characterized in that it comprises a. Receiving, by an image processing server, candidate image data and image data included in the search query from a search server which has performed a search using a search query received from a mobile terminal; Comparing, by the image processing server, a feature vector of the candidate image data with image data included in the search query; Calculating, by the image processing server, the similarity degree of each image data according to the comparison result and transmitting the calculated degree of similarity to the search server; Search service providing method using a search query, characterized in that it comprises a. The mobile terminal entering an image search menu; Transmitting, by the portable terminal, a search query generated from text data and image data to a search server; Outputting, by the portable terminal, a search list for the search query received from the search server; Transmitting, by the portable terminal, the selection signal to the search server when a selection signal for at least one of the search lists is input; The mobile terminal receiving and outputting image data corresponding to the selection signal from the search server; Search service providing method using a search query, characterized in that it comprises a.

Images