KR102178177B1 - Terminal and service providing device, control method thereof, computer readable medium having computer program recorded therefor and image searching system - Google Patents

Abstract

The present invention discloses a terminal, a service providing apparatus, a control method thereof, a recording medium in which a computer program is recorded, and an image search system. That is, the present invention accurately provides information on the photographing object by using the reference image for the photographing object and the photographing attitude for the photographing object, and when the photographing attitude of the camera for the photographing object to be searched is changed, the changed photographing screen User convenience can be improved by quickly providing search results for.

Description

Terminal and service providing device, control method thereof, computer readable medium having computer program recorded therefor and image searching system}

The present invention relates to a terminal, a service providing apparatus, a method for controlling the same, a recording medium in which a computer program is recorded, and an image search system. In particular, a terminal capable of accurately searching for a photographing object by using a photographing posture of a camera with respect to the photographing object, It relates to a service providing apparatus, a control method thereof, a recording medium in which a computer program is recorded, and an image search system.

Portable electronic devices are electronic devices that are portable and have one or more functions such as voice and video call functions, functions to input and output information, and functions to store data.

As the functions of portable electronic devices are diversified, they are implemented in the form of a multimedia player with complex functions such as, for example, taking pictures or videos, playing music or video files, receiving games, broadcasting, and connecting to the Internet. Has become.

Various new attempts in hardware or software are applied to such multimedia devices to implement complex functions.

On the other hand, with the recent development of communication technology and the activation of the Internet, various types of search have been conducted.

For example, in order to find information they want to know, users can search for the information they want by searching through the Internet, in addition to searching for a book or asking someone who is knowledgeable about the information.

That is, users can receive a search service linked with a web server by accessing a web server providing a search service using a web browser and then inputting and transmitting a keyword related to the information to be found.

Moreover, recently, a service for searching for the same or similar images by a search service has been provided even when the desired information is composed of images.

However, even in this case, when the user does not know what the image is, there is a limitation in providing information about the image accurately. In addition, there is a limitation in that the user is provided with images similar to the image corresponding to the search target.

Korean Patent Registration No. 10-1307341 [Name: Dynamic object motion capture method and device]

The present invention provides a terminal capable of accurately providing information on a photographing object by using an image of a photographing object and a photographing posture of a camera with respect to the photographing object, a service providing apparatus, a control method thereof, a recording medium and an image in which a computer program is recorded. We want to provide a search system.

In addition, the present invention is a terminal capable of quickly providing a search result for the changed photographing screen, a service providing device, a control method thereof, and a computer program in which a camera's photographing posture for a target to be searched is changed. It is intended to provide a recording medium and image retrieval system.

In addition, the present invention provides a terminal, a service providing device, a control method thereof, a recording medium in which a computer program is recorded, and an image search system capable of increasing search data and improving search accuracy based on a user reply to a search result. I want to.

A terminal according to an embodiment of the present invention includes an image acquisition unit for acquiring a first image including first imaging data and a second image including second imaging data; And a control unit that obtains search candidate data from the comparison data based on the comparison data and the first image, and obtains a derived image from the search candidate data based on the search candidate data, the first photographing data, and the second photographing data. have.

An apparatus for providing a service according to an exemplary embodiment of the present invention includes an image acquisition unit that acquires a first image with first photographing data and a second image with second photographing data; And a control unit that obtains search candidate data from the comparison data based on the comparison data and the first image, and obtains a derived image from the search candidate data based on the search candidate data, the first photographing data, and the second photographing data. have.

As an example related to the present invention, the photographing data may include at least one of tilt information and orientation information.

As an example related to the present invention, the comparison data may include a reference image, a derivative image of each reference image, and photographing data of each image.

As an example related to the present invention, the control unit may include a first comparison module that compares the reference image of the comparison data and the first image.

As an example related to the present invention, the first comparison module may acquire a preset number of search candidate data in order of high similarity based on the comparison result.

As an example related to the present invention, the control unit includes: a photographing data calculation module for obtaining changed photographing data based on first photographing data and second photographing data; And a second comparison module for comparing the photographing data included in the search candidate data with the changed photographing data.

As an example related to the present invention, the second comparison module may acquire a preset number of derived images in the order of high similarity based on the comparison result.

As an example related to the present invention, the controller may acquire a plurality of derived images.

A method for controlling a service providing apparatus according to an embodiment of the present invention includes: obtaining a first image including first photographing data from a terminal through an image acquisition unit; Obtaining search candidate data from the comparison data based on the comparison data and the first image through a control unit; Acquiring a second image including second photographing data from a terminal through an image acquisition unit; Obtaining a plurality of derived images from the search candidate data based on the search candidate data, the first photographing data, and the second photographing data through a control unit; And transmitting a plurality of derived images and identification information to the terminal through the communication unit.

As an example related to the present invention, it may further include receiving a user reply for a plurality of derived images through a communication unit.

As an example related to the present invention, the storage unit may further include storing the first image, the second image, and comparison data generated based on the user reply.

A computer program for performing the control method of the service providing apparatus according to the above-described embodiment may be stored in the recording medium on which the computer program according to the embodiment of the present invention is recorded.

An image retrieval system according to an embodiment of the present invention includes a photographing unit that captures a first image and a second image, a sensing unit that acquires photographing data of the first image and the second image, and transmits the first image and the second image. Terminal; And an image acquisition unit that acquires a first image with first photographing data and a second image with second photographing data, and obtains search candidate data from the comparison data based on the comparison data and the first image, and It may include a service providing apparatus including a control unit for obtaining a derived image from the search candidate data based on the data, the first photographing data, and the second photographing data.

The present invention has the effect of accurately providing information on a photographing object by using a reference image for a photographing object and a photographing posture for the photographing object.

In addition, according to the present invention, when a photographing posture of a camera for a photographing object to be searched is changed, it is possible to quickly provide a search result for the changed photographing screen.

In addition, the present invention has the effect of increasing search data and improving search accuracy based on a user reply to a search result.

1 is a diagram showing the configuration of an image search system according to an embodiment of the present invention.
2 is a block diagram of a terminal according to an embodiment of the present invention.
3 is a block diagram of an apparatus for providing a service according to an embodiment of the present invention.
4 is a diagram illustrating an example of a search target, a first image, and a second image.
5 is a diagram illustrating an example of comparison data.
6 is an example of a result of obtaining a similarity between a first image and a reference image.
7 is an example of a result of obtaining a similarity between changed photographing data and derived photographing data.
8 is a flowchart illustrating a method of controlling a service providing apparatus according to an embodiment of the present invention.
9 is a diagram illustrating an example of a search result displayed on a terminal.

It should be noted that the technical terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be interpreted as generally understood by those of ordinary skill in the technical field to which the present invention belongs, unless otherwise defined in the present invention, and is excessively comprehensive. It should not be construed as a human meaning or an excessively reduced meaning. In addition, when a technical term used in the present invention is an incorrect technical term that does not accurately express the spirit of the present invention, it should be replaced with a technical term that can be correctly understood by those skilled in the art. In addition, general terms used in the present invention should be interpreted as defined in the dictionary or according to the context before and after, and should not be interpreted as an excessively reduced meaning.

In addition, the singular expression used in the present invention includes a plurality of expressions unless the context clearly indicates otherwise. In the present invention, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various elements or various steps described in the invention, and some of the elements or some steps are included. It should be construed that it may not be, or may further include additional components or steps.

In addition, terms including ordinal numbers such as first and second used in the present invention may be used to describe the constituent elements, but the constituent elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of the reference numerals, and redundant descriptions thereof will be omitted.

In addition, in describing the present invention, when it is determined that a detailed description of a related known technology may obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

The terminal described herein is a concept generically referring to all types of terminals that perform a selected task by receiving a user's command such as a tablet PC, a notebook computer, and a desktop PC, as well as a handheld terminal such as a mobile communication terminal.

The service providing apparatus described herein is a concept generically referring to all kinds of electronic devices that store data related to a service requested from a terminal, transmit and receive data to and from the terminal, and support the operation of an application provided by the terminal.

1 is a diagram showing the configuration of an image search system according to an embodiment of the present invention.

As illustrated in FIG. 1, the image search system may include a terminal 200 and a service providing device 300. Not all components of the image retrieval system shown in FIG. 1 are essential components, and an image retrieval system may be implemented by more components than those shown in FIG. 1, and image retrieval by fewer components. The system may be implemented.

The terminal 200 may acquire a plurality of images (first image and second image) of the target 100 to be searched. In addition, the terminal 200 may transmit a plurality of acquired images to the service providing apparatus 300.

The service providing apparatus 300 may obtain search candidate data from comparison data provided based on the first image. Here, the search candidate data may mean an object to be searched among the comparison data.

Also, the service providing apparatus 300 may acquire an image having a composition similar to that of the second image from the search candidate data based on the first photographing data included in the first image and the photographing data included in the second image.

Here, the photographing data means the inclination and orientation of the imaging device when an image is captured.

The service providing apparatus 300 may provide the acquired image to the terminal 200. The user can know information about the first image and the second image through the image displayed on the terminal 200. For example, if the user does not know the name of the search target, the user can check the name of the search target through the search result displayed on the terminal 200.

2 is a block diagram of a terminal 200 according to an embodiment of the present invention.

As shown in FIG. 2, the terminal 200 may include a photographing unit 210, a transmission unit 220, a receiving unit 230, a display unit 240, a user input unit 250, and a detection unit 260. have. The terminal 200 may be implemented by more components than those shown in FIG. 2, or the terminal 200 may be implemented by fewer components than that shown in FIG. 2.

The photographing unit 210 may capture an image of the target 100 to be searched. Meanwhile, the photographing unit 210 may capture a plurality of images (eg, a first image and a second image) of the target 100 to be searched. For example, the photographing unit 210 may capture an image such as a front view, a rear view, a left view, a right view, and a perspective view of the target 100 to be searched.

The display unit 240 may display an image of the search target when the photographing unit 210 captures an image of the search target. Also, the display unit 240 may display a search result including an image similar to the search target image.

The sensing unit 260 includes a tilt sensor for measuring the tilt of the terminal 200, an orientation sensor for measuring the orientation of the terminal 200, and the like, and may detect the tilt and orientation of the terminal. For example, the sensing unit 260 of the terminal may measure the degree of inclination of the terminal and the direction the terminal is looking at. In addition, when the photographing unit 210 captures the search target and acquires the first image and the second image, the tilt information and the orientation information of the terminal measured by the sensing unit 260 are added to the first image and the second image. Have to be.

In this case, information added to the first image will be defined as first photographing data. In addition, information added to the second image will be defined as second photographing data.

As such, the photographing data may include tilt information and orientation information of the imaging device.

The user input unit 250 may obtain a user input.

The transmitter 220 may transmit data processed by the terminal 200 to a service providing device. For example, the transmitter 220 may transmit the image acquired by the photographing unit 210 to the service providing device.

The receiving unit 230 may receive data processed by the service providing apparatus 100. For example, the receiving unit 230 may receive a derived image from the service providing device 200.

3 is a block diagram of a service providing apparatus 300 according to an embodiment of the present invention.

As shown in FIG. 3, the service providing apparatus 300 may include an image acquisition unit 310, a control unit 320, a storage unit 330, and a communication unit 340. The service providing apparatus 300 may be implemented by more constituent elements than the constituent elements illustrated in FIG. 3, and the service providing apparatus 300 may be implemented by fewer constituent elements.

The image acquisition unit 310 may acquire a first image and a second image from the terminal. In this case, the first image may include first photographing data, and the second image may include second photographing data.

4 is a diagram illustrating an example of a search target, a first image, and a second image.

4A shows an example of a search target.

Fig. 4(b) shows a first image of a search target. Further, Fig. 4(c) shows a second image for the search target.

For example, the first image 110 may be a front view of a search target. In addition, the first photographing data 112 may include tilt information and orientation information of the imaging device when the first image is captured. Also, the second image 120 may be a top view of the search target. Further, the second photographing data 122 may include tilt information and orientation information of the imaging device when the second image is captured.

For convenience of explanation, in the present embodiment, it has been described that the first image and the second image are acquired. However, when an image of a search target is captured by capturing a video, more images may be continuously acquired.

The storage unit 330 may store comparison data.

Here, the comparison data is used to search for an image similar to the search target, and may include a reference image, a derived image of each reference image, and photographing data of each image.

Here, the reference image may mean an image to be compared with the first image. In addition, the derived image of each reference image may mean an image captured from a different angle by making a composition different from that of the reference image.

5 is a diagram illustrating an example of comparison data.

As shown in FIG. 5, the comparison data includes identification information (440-1, 440-2, 440-3) representing each object, and reference images (410-1, 410-2, 410-3) representing each object. It can be provided.

In addition, a plurality of derived images 420-1, 420-2, and 420-3 may be provided for each reference image 410-1.

In addition, the comparison data includes shooting data (430-1, 430-2, 430-3) of each image (410-1, 420-1, 410-2, 420-2, 410-3, 420-3) can do.

For example, the comparison data includes identification information 440-1 of the first object, a reference image 410-1 representing the first object, and a plurality of derived images captured at a different slope and direction than the reference image 410-1 ( 420-1), and the photographing data 430-1 of each image 410-1 and 420-1 may be included.

Here, the reference image 410-1 may be an image in which the front of the first object is captured, the first derived image 422 may be an image in which the top of the first object is captured, and the second derived image 424 ) May be an image in which the right side of the first object is imaged, and the third derived image 426 may be an image in which the diagonal surface of the first object is imaged.

Meanwhile, here, the reference image and the derived image are not fixed. For example, when the first derived image is used as the reference image, the reference image may be used as the derived image.

The controller 320 may obtain search candidate data from the comparison data based on the comparison data and the first image. Here, the search candidate data may mean an object to be searched among the comparison data. Also, the controller 320 may obtain a derived image from the search candidate data based on the search candidate data, the first photographing data, and the second photographing data.

The control unit 320 may include a first comparison module 322, a photographing data calculation module 324, and a second comparison module 326.

4 and 5, the first comparison module 322 of the controller compares the reference images 410-1, 410-2, and 410-3 of the comparison data with the first image 110, and A similarity between 110 and each of the reference images 410-1, 410-2, and 410-3 may be obtained.

6 is an example of a result of obtaining a similarity between a first image and each reference image.

As shown in FIG. 6, the first comparison module 322 of the controller may obtain a similarity 530 between the first image and each reference image.

In this case, the first comparison module 322 may acquire a preset number of search candidate data in the order of high similarity based on the comparison result. For convenience of explanation, in the present embodiment, it is assumed that one search candidate data is obtained.

Referring to FIG. 6, it can be seen that the degree of similarity between the first image and the first reference image is highest. Accordingly, the first comparison module 322 may set the content of the first reference image as the search candidate data 500. That is, the first comparison module 322 searches for the identification information 440-1, the reference image 410-1, the derived image 420-1, and the photographing data 430-1 of the first object. Can be set to

The photographing data calculation module 324 may obtain changed photographing data based on the first photographing data and the second photographing data.

Specifically, the photographing data calculation module 324 compares the inclination information of the imaging device when the first image is captured, the orientation information and the inclination information of the imaging device when the second image is captured, and the change shooting data. Can be obtained.

That is, the modified photographing data may indicate a tilt and azimuth of the imaging device when a second image is captured based on a tilt and azimuth of the imaging device when the first image is captured. In addition, the changed photographing data may indicate a degree of change in posture of the imaging device when the first image and the second image are captured.

Referring to FIG. 4, the photographing data operation module may acquire changed photographing data based on the first photographing data 112 and the second photographing data 122.

The second comparison module 326 of the controller compares the derived photographing data 423, 425, and 427 included in the search candidate data with the changed photographing data, and the derived photographing data 423 provided in the changed photographing data and the search candidate data. 425, 427) can be obtained. Here, the derived photographing data may mean photographing data of a derived image.

7 is an example of a result of obtaining a similarity between changed photographing data and derived photographing data.

As shown in FIG. 7, the second comparison module 326 of the control unit may obtain a similarity between the modified photographing data and the derived photographing data.

In this case, the second comparison module 326 may acquire a preset number of derived images in the order of high similarity based on the comparison result. Specifically, the second comparison module 326 may acquire a derived image corresponding to the derived photographing data based on the comparison result.

Referring to FIG. 7, it can be seen that the similarity with the changed photographing data is high in the order of the first derived photographing data, the third derived photographing data, and the second derived photographing data.

In this case, the control unit may acquire one derived image. Alternatively, the controller may acquire a plurality of derived images.

When the photographing posture of the terminal for the search target is changed, when the terminal has a plurality of derived images, the terminal can quickly display the search result.

The communication unit 340 may transmit the derived image obtained by the control unit and identification information on the derived image to the terminal.

As described above, the present invention has an effect that information on a photographing object can be accurately provided by using an image of a photographing object and a photographing posture of the photographing object.

8 is a flowchart illustrating a method of controlling a service providing apparatus according to an embodiment of the present invention.

Referring to FIG. 8, the service providing apparatus 200 may acquire a first image including first photographing data from a terminal (S810).

In addition, the service providing apparatus may obtain search candidate data based on the comparison data and the obtained first image (820 ).

In addition, the service providing apparatus may acquire a second image including second photographing data from the terminal (S830).

Also, the service providing apparatus may acquire a plurality of derived images based on the search candidate data, the first photographing data, and the second photographing data (S840).

Specifically, the service providing device 300 acquires the change photographing data based on the first photographing data and the second photographing data, and a plurality of derived images based on the obtained change photographing data and photographing data provided in the search candidate data. Can be obtained.

In this case, the service providing device may acquire not only the derived image having the highest similarity with the changed photographing data, but also the derived image of the difference order of a preset number.

In addition, the service providing apparatus may transmit the acquired plurality of derived images and identification information thereof to the terminal (S850).

Accordingly, the terminal may obtain a search result (derived image) in a posture adjacent to the first posture as well as a search result similar to the image captured by the UE in the first posture from the service providing device. Therefore, even if the photographing posture of the terminal is changed, the terminal can quickly display the search result (derived image).

9 is a diagram illustrating an example of a search result displayed on a terminal.

As shown in FIG. 9, the search result may include identification information 440.

Accordingly, the user can find out information on the object to be searched through the identification information.

In addition, the terminal may transmit a user reply indicating an object selected by the user among the displayed search results. That is, the service providing apparatus may receive a user reply for the search result from the terminal (S860).

In this case, the service providing device may generate comparison data based on the first image, the second image, and the user reply (S870). In addition, the storage unit of the service providing device may store the generated comparison data.

Specifically, the service providing apparatus may add the first image, the second image, and identification information selected by the user reply to the comparison data.

In this way, since the comparison data may be accumulated according to the user's search result, search accuracy may be greatly improved.

Meanwhile, it has been described above that similar images can be searched by the terminal and the service providing device. However, when a configuration provided in the service providing device is provided in the terminal, an image search method may be provided by the terminal.

A method of controlling a terminal and a service providing apparatus according to an embodiment of the present invention can be written in a computer program, and codes and code segments constituting a computer program can be easily inferred by a computer programmer in the art. In addition, the computer program is stored in a computer readable media, and is read and executed by a computer, a terminal according to an embodiment of the present invention, and a service providing device, thereby controlling the terminal and the service providing device. The method can be implemented.

Information storage media include magnetic recording media, optical recording media, and carrier wave media. A computer program implementing a method for controlling a terminal and a service providing device according to an embodiment of the present invention may be stored and installed in an internal memory such as a terminal, a service providing device, and an affiliate server. Alternatively, an external memory such as a smart card storing and installing a computer program for implementing a method for controlling a terminal and a service providing device according to an embodiment of the present invention may be mounted on the terminal or the service providing device through an interface.

The above contents may be modified and modified without departing from the essential characteristics of the present invention by those of ordinary skill in the technical field to which the present invention belongs. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

The present invention accurately provides information on a photographing object using a reference image for a photographing object and a photographing attitude for the photographing object, and when the photographing attitude of a camera for the photographing object to be searched is changed, By providing search results quickly to improve user convenience, it can be widely used in the field of e-commerce and marketing.

200: terminal 210: photographing unit
220: transmitting unit 230: receiving unit
240: display unit 250: user input unit
260: sensing unit 300: service providing device
310: image acquisition unit 320: control unit
330: storage unit 340: communication unit

Claims (14)

delete An image acquisition unit that acquires a first image with first photographing data and a second image with second photographing data; And
It includes identification information for each object, a reference image for each object, a plurality of derived images for each object, and comparison data including photographing data of each image, and compares the reference image for each object with the first image When a first reference image having the highest similarity to one image is obtained from the comparison data, identification information of a first object related to the first reference image, the first reference image, and a plurality of the first objects are obtained from the comparison data. Inclination of the imaging device in the case of setting search candidate data including a derived image of and photographing data for each image of the first object, and photographing the first image based on the first photographing data and the second photographing data And obtaining changed photographing data indicating a tilt and azimuth of the imaging device when the second image is photographed based on the orientation, and photographing data for each of the plurality of derivative images provided in the search candidate data. Acquires a preset number of derived images in the order of high similarity among a plurality of derived images with respect to the first object based on the comparison result compared to, and identifies an object with respect to the acquired derived image and the obtained derived image A service providing device including a control unit that provides information. delete delete delete delete delete delete delete Acquiring a first image including first photographing data from a terminal through an image acquisition unit;
The first image and the first image are compared with the first image by comparing identification information for each object, the reference image for each object, and comparison data including the plurality of derived images for each object and photographing data for each image through the control unit. A high first reference image is obtained from the comparison data, and identification information of a first object associated with the first reference image from the comparison data, the first reference image, a plurality of derived images for the first object, and the first 1 setting search candidate data including photographing data for each image of an object;
Acquiring a second image including second photographing data from the terminal through an image acquisition unit;
Tilt and azimuth of the imaging device when the second image is captured based on a tilt and azimuth of the imaging device when the first image is captured based on the first and second shooting data through a control unit A degree of similarity among a plurality of derived images with respect to the first object is obtained based on a comparison result of comparing the changed photographing data showing the changed photographing data with the photographing data for each of the plurality of derived images included in the search candidate data. Obtaining a preset number of derived images in the highest order; And
And transmitting the acquired derived image and identification information of an object with respect to the acquired derived image to the terminal through a communication unit. delete delete delete The first image comprising a photographing unit for capturing a first image and a second image, and a sensing unit for obtaining photographing data of each of the first and second images, and to which first photographing data obtained through the sensing unit is added. A terminal for transmitting the second image to which the image and second photographing data acquired through the sensing unit are added; And
The first and second images are received, and include identification information for each object, a reference image for each object, a plurality of derived images for each object, and comparison data including photographing data for each image, and the reference for each object When the first reference image having the highest similarity with the first image is obtained from the comparison data by comparing the image with the first image, the identification information of the first object associated with the first reference image from the comparison data, the first Set search candidate data including a first reference image, a plurality of derived images for the first object, and photographing data for each image of the first object, and based on the first photographing data and the second photographing data, Changed photographing data indicating the tilt and orientation of the imaging device when the second image is captured is obtained based on the tilt and orientation of the imaging device when the 1 image is captured, and the modified photographing data is the search candidate data A preset number of derived images are obtained in the order of similarity among the plurality of derived images for the first object, based on a comparison result compared with the photographed data for each of the plurality of derived images provided in, and the obtained derived images And a service providing device that provides identification information of an object with respect to the obtained derived image to the terminal.

Images