KR102178172B1 - Terminal and service providing device, control method thereof, computer readable medium having computer program recorded therefor and image searching system - Google Patents

Abstract

The present invention discloses a terminal, a service providing apparatus, a control method thereof, a recording medium in which a computer program is recorded, and an image retrieval system. That is, the present invention can increase the accuracy of the image search result by accurately providing information on the object to be photographed and converting it into a database by using the reference image for the object to be photographed and the position of the camera for the object to be photographed.

Description

Terminal and service providing device, control method thereof, computer readable medium having computer program recorded therefor and image searching system}

The present invention relates to a terminal, a service providing apparatus, a method for controlling the same, a recording medium in which a computer program is recorded, and an image search system. In particular, a terminal capable of accurately searching for a photographing object by using a photographing posture of a camera with respect to the photographing object, It relates to a service providing apparatus, a control method thereof, a recording medium in which a computer program is recorded, and an image search system.

Portable electronic devices are electronic devices that are portable and have one or more functions such as voice and video call functions, functions to input and output information, and functions to store data.

As the functions of portable electronic devices are diversified, they are implemented in the form of a multimedia player with complex functions such as, for example, taking pictures or videos, playing music or video files, receiving games, broadcasting, and connecting to the Internet. Has become.

Various new attempts in hardware or software are applied to such multimedia devices to implement complex functions.

On the other hand, with the recent development of communication technology and the activation of the Internet, various types of search have been conducted.

For example, in order to find information they want to know, users can search for the information they want by searching through the Internet, in addition to searching for a book or asking someone who is knowledgeable about the information.

That is, users can receive a search service linked with a web server by accessing a web server providing a search service using a web browser and then inputting and transmitting a keyword related to the information to be found.

Moreover, recently, a service for searching for the same or similar images by a search service has been provided even when the desired information is composed of images.

However, even in this case, when the user does not know what the image is, there is a limitation in providing information about the image accurately. In addition, there is a limitation in that the user is provided with images similar to the image corresponding to the search target.

Korean Patent Registration No. 10-1307341 [Name: Dynamic object motion capture method and device]

The present invention provides a terminal capable of accurately providing information on a photographing object by using a reference image for a photographing object and a photographing posture of a camera for the photographing object, a service providing apparatus, a control method thereof, a recording medium in which a computer program is recorded, and We want to provide an image search system.

In addition, an object of the present invention is to provide a terminal, a service providing apparatus, a control method thereof, a recording medium in which a computer program is recorded, and an image retrieval system capable of providing to a user by arranging images similar to a subject in the order of similarity.

In addition, the present invention provides a terminal, a service providing apparatus, a control method thereof, a recording medium in which a computer program is recorded, and an image search system capable of increasing search data and improving search accuracy based on a user reply to a search result. I want to.

A terminal according to an embodiment of the present invention includes an image acquisition unit that obtains a search request image and a search reference image; A photographing data management unit that acquires search photographing data including at least one of tilt information and orientation information based on the search request image and the search reference image; A search unit for obtaining a search result based on the comparison data, the search reference image, and search photographing data; And a display unit that displays the search result.

An apparatus for providing a service according to an embodiment of the present invention includes an image acquisition unit that acquires a first image and a second image from a terminal; A reference image management unit that sets one of the first image and the second image as a search reference image and sets the other as a search request image; A photographing data management unit that acquires search photographing data including at least one of tilt information and azimuth information based on the search reference image and the search request image; A search unit for obtaining a search result based on the comparison data, the search reference image, and search photographing data; And it may include a communication unit for transmitting the search result to the terminal.

As an example related to the present invention, the reference image management unit may extract a feature part of a search request target and set an image including the feature part as a search reference image.

As an example related to the present invention, the comparison data may include a comparison reference image and comparison photographing data of each comparison reference image.

As an example related to the present invention, the search unit may compare the search reference image and the comparison reference image, and compare the search photographing data and the comparison photographing data based on the comparison result.

As an example related to the present invention, the search unit may obtain a preset number of search results in the order of a high degree of similarity as a result of comparison.

As an example related to the present invention, the communication unit may receive a user reply for a search result from the terminal.

As an example related to the present invention, a storage unit for storing comparison data may be further included.

As an example related to the present invention, the storage unit may store a search reference image, a search request image, search photographing data, and comparison data generated based on a user reply.

As an example related to the present invention, the search result may include target identification information.

A method of controlling a service providing apparatus according to an embodiment of the present invention includes: obtaining a first image and a second image from a terminal through an image acquisition unit; Setting one of the first image and the second image as a search reference image and setting the other as a search request image through a reference image management unit; Obtaining search photographing data including at least one of tilt information and orientation information based on a search reference image and a search request image through a photographing data management unit; Obtaining a search result based on the comparison data, the search reference image, and the search photographing data through a search unit; And transmitting the search result to the terminal through the communication unit.

A computer program for performing the control method of the service providing apparatus according to the above-described embodiment may be stored in the recording medium on which the computer program according to the embodiment of the present invention is recorded.

An image search system according to an embodiment of the present invention includes: a photographing unit for capturing a first image and a second image, and a terminal for transmitting the first and second images; And an image acquisition unit receiving the first image and the second image from the terminal, a reference image management unit configured to set any one of the first image and the second image as a search reference image and set the other as a search request image, a search reference A photographing data management unit that obtains search photographing data including at least one of tilt information and azimuth information based on an image and a search request image, a search unit that obtains a search result based on comparison data, a search reference image, and search photographing data, and It may include a service providing apparatus including a communication unit for transmitting the search result to the terminal.

The present invention has the effect of accurately providing information on a photographing object by using a reference image for a photographing object and a photographing posture for the photographing object.

In addition, the present invention has an effect that images similar to the subject to be photographed can be arranged in order of similarity and provided to the user.

In addition, the present invention has the effect of increasing search data and improving search accuracy based on a user reply to a search result.

1 is a diagram showing the configuration of an image search system according to an embodiment of the present invention.
2 is a block diagram of a terminal 200 according to an embodiment of the present invention.
3 is a block diagram of a service providing apparatus 300 according to an embodiment of the present invention.
4 is a diagram illustrating an example of a search request image, a search reference image, and search photographing data.
5 is a diagram illustrating an example of comparison data.
6 is a diagram illustrating an example of a plurality of comparison reference images included in comparison data.
7 is a diagram illustrating an example of a first search result.
8 is a diagram illustrating an example of a plurality of comparative photographing data belonging to a comparison reference image.
9 is a diagram illustrating an example of a second search result.
10 is a flowchart illustrating a method of controlling a service providing apparatus according to an embodiment of the present invention.
11 is a diagram illustrating an example of a search result displayed on a terminal.

It should be noted that the technical terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be interpreted as generally understood by those of ordinary skill in the technical field to which the present invention belongs, unless otherwise defined in the present invention, and is excessively comprehensive. It should not be construed as a human meaning or an excessively reduced meaning. In addition, when a technical term used in the present invention is an incorrect technical term that does not accurately express the spirit of the present invention, it should be replaced with a technical term that can be correctly understood by those skilled in the art. In addition, general terms used in the present invention should be interpreted as defined in the dictionary or according to the context before and after, and should not be interpreted as an excessively reduced meaning.

In addition, the singular expression used in the present invention includes a plurality of expressions unless the context clearly indicates otherwise. In the present invention, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various elements or various steps described in the invention, and some of the elements or some steps are included. It should be construed that it may not be, or may further include additional components or steps.

In addition, terms including ordinal numbers such as first and second used in the present invention may be used to describe the constituent elements, but the constituent elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of the reference numerals, and redundant descriptions thereof will be omitted.

In addition, in describing the present invention, when it is determined that a detailed description of a related known technology may obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

The terminal described herein is a concept generically referring to all types of terminals that perform a selected task by receiving a user's command such as a tablet PC, a notebook computer, and a desktop PC, as well as a handheld terminal such as a mobile communication terminal.

The service providing apparatus described herein is a concept generically referring to all kinds of electronic devices that store data related to a service requested from a terminal, transmit and receive data to and from the terminal, and support the operation of an application provided by the terminal.

1 is a diagram showing the configuration of an image search system according to an embodiment of the present invention.

As illustrated in FIG. 1, the image search system may include a terminal 200 and a service providing device 300. Not all components of the image retrieval system shown in FIG. 1 are essential components, and an image retrieval system may be implemented by more components than those shown in FIG. 1, and image retrieval by fewer components. The system may be implemented.

The terminal 200 may acquire a plurality of images (first image and second image) of the target 100 to be searched. In addition, the terminal 200 may transmit a plurality of acquired images to the service providing apparatus 300.

The service providing device may set one of the first image and the second image as a search reference image and set the other as a search request image.

Here, the search reference image is an image used as a reference for comparing with data provided in the service providing device. The search reference image may be an image in which features of a search target are well displayed. Alternatively, the search reference image may be a front view of the search target. In this way, the search reference image may be appropriately set according to the needs of the user.

The service providing apparatus 300 may acquire search photographing data based on the search request image and the search reference image.

The search photographing data may mean a tilt and azimuth of the imaging device when the search reference image is captured, based on the orientation, and when the search request image is captured.

The service providing apparatus 300 may obtain a search result including an image similar to the search request image based on the comparison data, search reference image, and search photographing data provided in advance.

The service providing apparatus 300 may provide the obtained search result to the terminal 200. The user can know information about the search request image through the search result displayed on the terminal 200. For example, if the user does not know the name of the search target, the user can check the name of the search target through the search result displayed on the terminal 200.

2 is a block diagram of a terminal 200 according to an embodiment of the present invention.

As shown in FIG. 2, the terminal 200 may include a photographing unit 210, a transmission unit 220, a receiving unit 230, a display unit 240, a user input unit 250, and a detection unit 260. have. The terminal 200 may be implemented by more components than those shown in FIG. 2, or the terminal 200 may be implemented by fewer components than that shown in FIG. 2.

The photographing unit 210 may capture an image of the target 100 to be searched. Meanwhile, the photographing unit 210 may capture a plurality of images (eg, a first image and a second image) of the target 100 to be searched. For example, the photographing unit 210 may capture an image such as a front view, a rear view, a left view, a right view, and a perspective view of the target 100 to be searched.

The display unit 240 may display an image of the search target when the photographing unit 210 captures an image of the search target. In addition, the display unit 240 may display a search result including an image similar to the search request image.

The sensing unit 260 includes a tilt sensor for measuring the tilt of the terminal 200, an orientation sensor for measuring the orientation of the terminal 200, and the like, and may detect the tilt and orientation of the terminal. For example, the sensing unit 260 of the terminal may measure the degree of inclination of the terminal and the direction the terminal is looking at. In addition, when the photographing unit 210 captures the search target and acquires the first image and the second image, the tilt information and the orientation information of the terminal measured by the sensing unit 260 are added to the first image and the second image. Have to be.

The user input unit 250 may obtain a user input.

The transmitter 220 may transmit data processed by the terminal 200 to a service providing device. For example, the transmitter 220 may transmit the image acquired by the photographing unit 210 to the service providing device.

The receiving unit 230 may receive data processed by the service providing apparatus 100. For example, the receiving unit 230 may receive a search result from the service providing device 200.

3 is a block diagram of a service providing apparatus 300 according to an embodiment of the present invention.

As shown in FIG. 3, the service providing device 300 includes an image acquisition unit 310, a reference image management unit 320, a photographing data management unit 330, a search unit 340, a storage unit 350, and a communication unit ( 360). The service providing apparatus 300 may be implemented by more constituent elements than the constituent elements illustrated in FIG. 3, and the service providing apparatus 300 may be implemented by fewer constituent elements.

The image acquisition unit 310 may acquire a first image and a second image from the terminal.

The reference image management unit 320 may set one of the first image and the second image as a search reference image and set the other as a search request image.

Here, the search reference image is an image used to compare with data provided in the service providing device.

The search reference image may be an image in which features of the search target are well displayed. Here, the feature part of the search request image is a part that clearly indicates the search target and may be appropriately set as necessary.

Also, the search reference image may be a front view of the search target.

The method of setting the search reference image may be appropriately selected according to the needs of the user.

In addition, the search request image may be an image indicating the shape (position) of an object expressed in the search result.

Also, a search reference image and a search request image may be set based on a user input.

4 is a diagram illustrating an example of a search request image, a search reference image, and search photographing data.

4(a) shows an example of a search request image.

4(b) shows an example of a search reference image.

For example, the reference image management unit 320 includes a second image (a front view of a'search target') of a first image (a perspective view of a'search target') and a second image (a front view of a'search target') Is extracted as a feature part of a search target, and a second image may be set as a search reference image. In addition, the reference image management unit 320 may set the first image (a perspective view of the'search target') as a search request image.

That is, when the search request image and the search reference image are set as described above, the service providing apparatus 200 may obtain the comparison data group by the second image (a front view of the'search target'). In addition, the service providing apparatus 200 may obtain a search result based on the first image (a perspective view of the'search target'). That is, the service providing apparatus 200 may obtain a search result including a perspective view of a search target.

The photographing data management unit 330 may acquire search photographing data based on the search request image and the search reference image.

Here, the search photographing data may mean tilt information of the imaging device when capturing a search reference image, and tilt information of the imaging device when capturing a search request image, and azimuth information based on the orientation information. .

4C shows an example of search photographing data.

As shown in FIG. 4(c), search photographing data may include tilt information and orientation information.

At this time, the photographing data management unit 330 captures tilt information and orientation information of the imaging device when the search reference image shown in FIG. 4(b) is captured, and when the search request image shown in FIG. 4(a) is captured. Search photographing data may be obtained by comparing the tilt information and the orientation information of the imaging device of.

The storage unit 350 may store comparison data.

Here, the comparison data is used to search for an image similar to the search target, and may include a search reference image and an image used to compare with the comparison reference image.

The comparison data may include a comparison reference image and comparison photographing data of each comparison reference image.

Here, the comparison reference image refers to an image to be compared with the search reference image.

In addition, the comparative photographing data refers to search photographing data and data to be compared.

5 is a diagram illustrating an example of comparison data.

As shown in FIG. 5, the comparison data may include a plurality of comparison reference images 510-1 and 510-2.

In addition, a plurality of comparison photographing data 520-1, 520-2, and 520-3 may be provided for each comparison reference image 510-1.

That is, the comparison data may include a comparison reference image and comparison photographing data of each comparison reference image.

6 is a diagram illustrating an example of a plurality of comparison reference images included in comparison data.

As shown in FIG. 6, the comparison data may include comparison reference images 512, 514, 516, and 518 for different objects, respectively.

In this case, each of the comparison reference images 512, 514, 516, and 518 may include target identification information 513, 515, 517, and 519.

The search unit 340 may obtain a search result based on the comparison data, a search reference image, and search photographing data.

Specifically, the search unit 340 of the service providing apparatus 300 may obtain a first search result by comparing the search reference image obtained by the reference image management unit 320 with the comparison reference image included in the comparison data.

For example, the search unit 340 of the service providing device 300 may compare the search reference image shown in FIG. 4(b) and the comparison reference image shown in FIG. 6 to obtain a similarity between the search reference image and each comparison image. I can.

7 is a diagram illustrating an example of a first search result.

As shown in FIG. 7, the search unit 340 of the service providing apparatus 300 may obtain a similarity between the search reference image and each comparison reference image.

In addition, the search unit 340 of the service providing device 300 may select a preset comparison reference image in the order of higher similarity among the search results 530, and compare the comparison photographing data of the selected comparison reference image with the search photographing data. have.

6 and 7, it can be seen that the comparison reference image 512 and the search reference image have the highest similarity. Accordingly, the search unit 340 of the service providing apparatus 300 may select the comparison reference image 512 and compare the comparison photographing data belonging to the comparison reference image 512 with the search photographing data.

8 is a diagram illustrating an example of a plurality of comparative photographing data belonging to a comparison reference image.

As shown in FIG. 8, each of the comparative photographing data 612, 614, 616, 618 is a corresponding image, object identification information 613, 615, 617, 619, tilt information, and orientation information 622, 624, 626, 628. ) Can be included.

The search unit 340 of the service providing apparatus 300 compares the comparison photographing data 612, 614, 616, and 618 belonging to the comparison reference image selected based on the first search result with the search photographing data, and compares the second search result. Can be obtained.

For example, the search unit 340 of the service providing device 300 compares the search photographing data shown in FIG. 4(c) with the comparative photographing data shown in FIG. 8 to obtain a similarity between the search photographing data and each comparative photographing data. can do.

9 is a diagram illustrating an example of a second search result.

As illustrated in FIG. 9, the search unit 340 of the service providing apparatus 300 may obtain a similarity between search photographing data and each comparative photographing data.

Through such a process, the search unit 340 of the service providing device may obtain a preset number of search results in the order of the highest similarity as a result of comparison.

Meanwhile, the communication unit 360 may transmit the search result obtained by the search unit 340 to the terminal 200.

10 is a flowchart illustrating a method of controlling a service providing apparatus according to an embodiment of the present invention.

Referring to FIG. 10, the service providing apparatus 200 may obtain a first image and a second image from a terminal (S110).

In addition, the service providing device may set one of the first image and the second image as a search reference image and set the other as a search request image (S120).

In addition, the service providing device may acquire search photographing data based on the set search request image and search reference image (S130).

In addition, the service providing apparatus may obtain a search result based on preset comparison data, the obtained search reference image, and the obtained search photographing data (S140).

Specifically, the service providing apparatus 300 may obtain a first search result by comparing the search reference image with the comparison reference image included in the comparison data. In addition, the service providing apparatus 300 may obtain a second search result by comparing comparison photographing data belonging to the comparison reference image corresponding to the first search result with the search photographing data.

In this way, through the process of obtaining the first search result, the search target group may be specifically limited, so that the target search speed may be increased. In addition, since the search is performed through the reference image, the accuracy of the search may be improved.

In addition, through the process of obtaining the second search result, an image captured at a shooting angle similar to a shooting angle to be searched may be searched. Therefore, user convenience can be greatly improved.

In addition, the service providing apparatus may transmit the obtained search result to the terminal (S150).

Meanwhile, the terminal receiving the search result from the service providing device may display the received search result.

In this case, the search result displayed on the terminal may be an image obtained at a shooting angle similar to the search request image.

Also, the search result may include object identification information.

11 is a diagram illustrating an example of a search result displayed on a terminal.

As shown in FIG. 11, the search result may include object identification information 613 and 615.

Accordingly, the user can find out information on the object to be searched through the object identification information.

In addition, the terminal may transmit a user reply indicating an object selected by the user among the displayed search results. That is, the service providing apparatus may receive a user reply for the search result from the terminal (S160).

In this case, the service providing device may generate comparison data based on a search reference image, a search request image, search photographing data, and a user reply. In addition, the storage unit of the service providing device may store the generated comparison data.

Specifically, the service providing apparatus may add a search reference image, a search request image, search photographing data, and target identification information selected by a user reply to the comparison data.

In this way, since the comparison data may be accumulated according to the user's search result, search accuracy may be greatly improved.

Meanwhile, it has been described above that similar images can be searched by the terminal and the service providing device. However, when a configuration provided in the service providing device is provided in the terminal, an image search method may be provided by the terminal.

A method of controlling a terminal and a service providing apparatus according to an embodiment of the present invention can be written in a computer program, and codes and code segments constituting a computer program can be easily inferred by a computer programmer in the art. In addition, the computer program is stored in a computer readable media, and is read and executed by a computer, a terminal according to an embodiment of the present invention, and a service providing device, thereby controlling the terminal and the service providing device. The method can be implemented.

Information storage media include magnetic recording media, optical recording media, and carrier wave media. A computer program implementing a method for controlling a terminal and a service providing device according to an embodiment of the present invention may be stored and installed in an internal memory such as a terminal, a service providing device, and an affiliate server. Alternatively, an external memory such as a smart card storing and installing a computer program for implementing a method for controlling a terminal and a service providing device according to an embodiment of the present invention may be mounted on the terminal or the service providing device through an interface.

The above contents may be modified and modified without departing from the essential characteristics of the present invention by those of ordinary skill in the technical field to which the present invention belongs. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

The present invention is capable of increasing the accuracy of image search results by accurately providing information on a subject to be photographed using a reference image for a subject and a photographing posture of a camera for the subject to be photographed and making a database It can be widely used in marketing fields and the like.

200: terminal 210: photographing unit
220: transmitting unit 230: receiving unit
240: display unit 250: user input unit
260: detection unit
300: service providing device 310: image acquisition unit
320: reference image management unit 330: photographing data management unit
340: search unit 350: storage unit
360: Ministry of Communications

Claims (13)

delete An image acquisition unit that acquires a first image and a second image for a search target from a terminal;
A reference image management unit that sets one of the first image and the second image as a search reference image and sets the other as a search request image;
Tilt information of the imaging device when capturing the search reference image based on the search reference image and the search request image, and tilt information of the imaging device when capturing the search request image based on the orientation information, and A photographing data management unit for obtaining search photographing data that is orientation information;
A storage unit configured to store comparison data including a plurality of comparison reference images each including object identification information for different objects and a plurality of comparison photographing data to be compared with the search photographing data for each of the plurality of comparison reference images;
Comparing the search reference image and the comparison reference image of the comparison data to obtain a comparison reference image having the highest similarity as a first search result, and belonging to the comparison reference image according to the first search result based on the comparison data, respectively A second image, object identification information, tilt information, and orientation information including one or more images obtained from a similar shooting angle as the search request image and target identification information by comparing a plurality of comparison photographing data including the orientation information and the search photographing data A search unit for obtaining a search result; And
Service providing apparatus comprising a communication unit for transmitting the second search result to the terminal. delete delete delete delete delete delete delete delete Obtaining a first image and a second image for a search target from a terminal through an image acquisition unit;
Setting one of the first image and the second image as a search reference image and the other as a search request image through a reference image management unit;
The imaging when the search request image is captured based on tilt information and orientation information of the imaging device when the search reference image is captured based on the search reference image and the search request image through the photographing data management unit. Obtaining search photographing data that is tilt information and orientation information of the device;
A comparison reference image of comparison data including a plurality of comparison reference images including target identification information for each different object through the search unit and a plurality of comparison photographing data to be compared with the search photographed data for each of the plurality of comparison reference images, and Comparing the search reference images to obtain a comparison reference image having the highest similarity as a first search result, and belonging to the comparison reference image according to the first search result based on the comparison data, each image, object identification information, and tilt information And comparing a plurality of comparison photographing data including orientation information with the search photographing data to obtain a second search result including one or more images obtained at a photographing angle similar to the search request image and target identification information. And
And transmitting the second search result to the terminal through a communication unit. delete A photographing unit for capturing a first image and a second image for a search target, and a terminal for transmitting the first image and the second image; And
Receive a first image and a second image from the terminal, set one of the first image and the second image as a search reference image, set the other as a search request image, and set the search reference image and the Search photographing data that is tilt information and orientation information of the imaging device when capturing the search request image based on the tilt information and orientation information of the imaging device when capturing the search reference image based on the search request image A comparison reference image of comparison data including a plurality of comparison reference images including target identification information for different objects, and a plurality of comparison photographing data that are to be compared with the search photographing data for each of the plurality of comparison reference images. The comparison reference image with the highest similarity is obtained as a first search result by comparing the search reference image with the search reference image, and an image, object identification information, and slope, respectively, belonging to the comparison reference image according to the first search result based on the comparison data The terminal by comparing a plurality of comparative photographing data including information and orientation information with the search photographing data to obtain a second search result including one or more images obtained at a photographing angle similar to the search request image and target identification information An image retrieval system including a service providing device that transmits to.

Images