KR101910825B1 - Method, apparatus, system and computer program for providing aimage retrieval model - Google Patents

Abstract

A method for performing an image search in a service server, the method comprising: generating a first descriptor vector by applying a first level codebook in the query image to a database image, An A step of extracting at least one candidate image similar to the image; Extracting a second descriptor vector by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor vector, A step B of estimating a subject area of the subject; C) estimating a subject area of the candidate image by applying the topic model to the candidate image; And a step D of calculating a degree of similarity between a subject area of the query image and a subject area of the candidate image and setting a similarity ranking of the candidate image according to the degree of similarity.

Description

METHOD, APPARATUS, SYSTEM AND COMPUTER PROGRAM FOR PROVIDING AIMAGE RETRIEVAL MODEL < RTI ID = 0.0 >

The present invention relates to an image retrieval method. More particularly, the present invention relates to a method and apparatus for providing an image retrieval method capable of efficiently operating in a large image database.

The expansion of digital imaging devices and the development of network environment are leading to a rapid increase in image data, and the search for image information through various mobile devices is increasing.

In this situation, where the size of the image database is rapidly increasing, the demand for a more convenient and accurate search method is increasing. There are two frameworks for image retrieval, one for text-based image retrieval and the other for content-based image retrieval.

Text-based image retrieval is performed by labeling all images in the database as keywords and searching for images on a label basis. However, there is a problem that the subject of the human being is intervened in the labeling process and the mathematical result can not be derived.

On the other hand, content-based image retrieval performs retrieval using visual similarity between images. In particular, content-based information retrieval can satisfy the user's convenience when the user does not know the keyword, which is effective.

However, the method of retrieving images based on content without separate text help such as keywords is generally performed by encoding the query image into one vector and indexing the same. This approach is efficient for extracting initial results.

However, according to the conventional search method of encoding an image into one vector, since geometric information about the position, size, and orientation of the object is insufficient, it is very sensitive to changes in closure, viewpoint change, There is a problem that the accuracy of the result is low.

An object of the present invention is to provide an image retrieval method capable of efficiently operating in a large image database.

More specifically, the present invention applies VT (Vocabulary Tree) at two levels in a visual search technology, extracts a group of database images (initial search list) similar to the query image at the first level, And a method for calculating the similarity rankings of images belonging to the group based on the similarity of the query image by applying the topic model at the second level.

A method for performing image search in a service server according to an embodiment of the present invention includes receiving a query image, generating a first descriptor vector by applying a first level codebook in the query image, comparing the first descriptor vector with a database image An A step of extracting at least one candidate image similar to the query image; Extracting a second descriptor vector by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor vector, A step B of estimating a subject area of the subject; C) estimating a subject area of the candidate image by applying the topic model to the candidate image; And a step D of calculating a degree of similarity between a subject area of the query image and a subject area of the candidate image and setting a similarity ranking of the candidate image according to the degree of similarity.

Further comprising: collecting a plurality of images and applying the first level codebook to the collected images to generate a first descriptor vector; Generating a second descriptor vector by applying the second level codebook to the collected image; And indexing the first descriptor vector and / or the second descriptor vector as an index value into the corresponding image. Further, the step C may include estimating a subject area of the candidate image by applying the topic model to the second descriptor vector of the candidate image. Further, the step D may include estimating the similarity between the candidate image and the query image by applying a weight to the descriptor of the subject area of the candidate image.

Meanwhile, in a service server according to an embodiment of the present invention, a method of performing an image search includes collecting a plurality of images, extracting a first descriptor by applying a first level codebook to the collected images, Constructing a database in which a vector for the descriptor is indexed to the image as an index value; Generating a first descriptor vector by applying the first level codebook in the query image and comparing the vector with an index value of the database image to generate a candidate image similar to the query image, Extracting; Extracting a second descriptor by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor, Estimating a region; And estimating a subject area of the candidate image by applying the topic model to a second descriptor extracted by applying the second level codebook to the candidate image.

The method may further include calculating a degree of similarity between a subject area of the query image and a subject area of the candidate image and setting a similarity ranking of the candidate image according to the degree of similarity.

According to another aspect of the present invention, there is provided an apparatus for performing image search, including: a database including at least one image; A communication unit for receiving a query image; A descriptor extracting unit for generating a first descriptor vector by applying a first level codebook in the query image and extracting a second descriptor vector by applying a second level codebook having a higher dimension than the first level codebook; Estimating a subject area of the query image by applying a topic model for distinguishing a subject area and a background area to the second descriptor vector of the query image, applying the topic model to the candidate image, A subject area estimator for estimating a subject area; And extracting at least one candidate image that is similar to the query image in comparison with the first descriptor vector and the database, calculating a similarity between a subject area of the query image and a subject area of the candidate image, And a degree-of-similarity calculation unit for setting a degree-of-similarity ranking of the candidate image.

Here, the descriptor extracting unit may generate the first descriptor vector by applying the first level codebook to the image included in the database, generate the second descriptor vector by applying the second level codebook, And an index unit for indexing the first descriptor vector and / or the second descriptor vector into the corresponding image using the index value.

Meanwhile, an apparatus for performing image search according to an embodiment of the present invention extracts a first descriptor by applying a first level code book in a plurality of images, and indexes the vector for the first descriptor as an index value into the corresponding image An image database; A communication unit for receiving a query image; And generating a first descriptor vector by applying the first level codebook in the query image, extracting at least one candidate image similar to the query image by comparing the vector with an index value of the database image, Extracting a second descriptor by applying a second level codebook having a higher dimension than the first level codebook in the image and applying a topic model for distinguishing the subject area and the background area to the second descriptor, And a controller for estimating a subject area of the candidate image by applying the topic model to a second descriptor extracted by applying the second level codebook to the candidate image.

Further, the control unit may calculate the similarity between the subject area of the query image and the subject area of the candidate image, and set the similarity ranking of the candidate image according to the similarity.

Meanwhile, in a service server according to an embodiment of the present invention, a computer program recorded on a computer-readable storage medium for performing image search includes a function of collecting a plurality of images; A function of building a database of the collected images; Upon receipt of the query image, the first level code book is extracted from the query image to extract a first descriptor, a vector for the first level descriptor is generated, and the vector and the index value of the database image are compared A function of extracting at least one candidate image similar to the query image; Extracting a second descriptor by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor, A function of estimating a region; Estimating a subject area of the candidate image by applying the topic model to the second descriptor of the candidate image; A degree of similarity between a subject area of the query image and a subject area of the candidate image is calculated and a similarity ranking of the candidate image is set according to the degree of similarity.

Furthermore, in a service server according to an embodiment of the present invention, a computer program recorded on a computer-readable storage medium for performing an image search includes a function of generating an image database; Receiving a query image, generating a first descriptor vector by applying a first level codebook in the query image, and extracting at least one candidate image similar to the query image by comparing with the database image; Extracting a second descriptor vector by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor vector, A function of estimating a subject area of the subject; Applying the topic model to the candidate image to estimate a subject area of the candidate image; And calculating a degree of similarity between a subject area of the query image and a subject area of the candidate image, and setting a similarity ranking of the candidate image according to the degree of similarity.

According to the present invention, image retrieval can be performed quickly and efficiently in a large image database. More specifically, according to the present invention, an object can be efficiently extracted from an image, and further, a database image including an object similar to an object of a query image in a large database can be retrieved quickly and accurately.

According to such a configuration, a similar image can be searched by only a small amount of network resources without relying on text, thereby being applicable to web-based applications and mobile-based applications.

Further, according to the present invention, it is possible to efficiently and accurately search for the same similarity between the image displayed on the client and the image stored in the database of the server. Therefore, you do not need to insert the tag for search into the image in advance, and you can automatically match similar images without reference to separate text.

1 is a flowchart for explaining a process of extracting a feature point from a query image and calculating a descriptor
2 is a flowchart for explaining a process of extracting a candidate group of a database image similar to a query image according to an embodiment of the present invention
3 is a flowchart illustrating a process of setting a ranking of an image candidate group by applying a topic model according to an embodiment of the present invention
4 is a diagram for explaining the configuration of an image retrieval framework according to an embodiment of the present invention;
5 is a diagram for explaining an example of applying a topic model in an image according to an embodiment of the present invention;
6 is a diagram for explaining the relationship between parameters and parameters applied to the SLTMB according to the embodiment of the present invention;
7 is a flowchart for explaining an example of a similar image retrieval method according to another embodiment of the present invention
8 is a block diagram for illustrating a configuration of a client for transmitting a query image and receiving a similar image according to an embodiment of the present invention
9 is a block diagram illustrating a configuration of a server operating as an image search engine according to an embodiment of the present invention.

It is to be understood that the present invention is not limited to the description of the embodiments described below, and that various modifications may be made without departing from the technical scope of the present invention. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

In the drawings, the same components are denoted by the same reference numerals. And in the accompanying drawings, some of the elements may be exaggerated, omitted or schematically illustrated. It is intended to clearly illustrate the gist of the present invention by omitting unnecessary explanations not related to the gist of the present invention. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Image retrieval can be utilized in various ways. For example, product information search.

For example, a user viewing a blog using a desktop may be curious about the name or price of a baby article displayed on a web page. In this case, according to the embodiment of the present invention, the user can capture the object in the device, query the captured image, access the image search server with the image similar to the baby goods contained in the query and the goods information linked with the similar image have.

To this end, an extension program for generating a query may be installed in a web browsing tool in the form of a plug-in, an ActiveX, a tool bar, or the like, or may be embedded in a specific application in the form of an SDK in a mobile device or may operate as an independent application.

The service server will retrieve an image similar to the image object contained in the query from the merchandise database and provide the search results to the user device.

Image retrieval, which is the basis of the above services, is rapidly increasing in search requests for mobile devices. The user of the mobile device acquires the information associated with the object to be photographed through the pre-established database search by photographing the object to be searched or utilizing the photographed image.

For example, in the case of a product search, a user may photograph a product to be searched and search it in a database to suppress additional information such as a product name, a price, a function, and a selling place. In addition, the service provider can plan a service that takes a building on the street and recognizes the building and provides information in the building or surrounding information.

However, in order to satisfy various service scenarios related to image retrieval, image retrieval technology should be robust to changes in lighting and environment, change in shooting time, ii) to be able to distinguish image features including different objects, and iii) It should be able to be applied to various applications. Iv) It should be structured hierarchically to support network environment and various terminals to ensure mutual compatibility. V) Support structure for efficiently searching large image data. And vi) hardware features that support high-speed feature extraction and retrieval.

However, conventional image retrieval encodes the query image into a single vector and indexes it to retrieve the database, and match and verify it. This approach is generally efficient for extracting initial results. However, according to the conventional search method of encoding an image into one vector, geometric information about the position, size, orientation, and the like of the object is insufficient, so that it is very sensitive to changes such as closing, There was a problem.

The present invention is intended to solve such a problem. To this end, the present invention includes a configuration for extracting characteristics of an image, generating an initial search list through an encoding phrase, and then providing a search result of a large-volume database by way of adjusting a ranking of the list via a topic modeling phrase can do.

More specifically, according to an embodiment of the present invention, a VT (Vocabulary Tree) for encoding a descriptor for a feature point of an image can be distinguished from a first level and a second level.

The first level VT can be formed by constructing a rough code book from the image to be searched by considering the values of the image descriptor as the concept of words in Bow (Back of Words). If the first level VT is applied to the query image and the database image, candidate groups can be generated according to the frequency of the feature, and an initial search list can be generated by the matching and verification method.

Further, the second level VT according to the embodiment of the present invention can be created in a manner of constructing a refined codebook constituting a finer structure reflecting the statistical characteristics of the geometry in the local area of the image descriptor.

According to an embodiment of the present invention, the second level VT is applied to an image included in a query image and an initial search list to calculate a similarity, and a topic model is applied to determine whether there is actual matching information for an initial search list, Or verification.

The topic model is intended to calculate a model verification probability by repeatedly verifying whether a topic model is used to distinguish between background and object in an image. The topic model can be learned by a machine learning method of extracting a morphological distribution and background distribution from a large number of images and distinguishing objects of the image.

According to the embodiment of the present invention, when the topic model is applied together with the encoding step, it is possible to perform efficient ranking adjustment to cope with a large change in the object shape of the image. Furthermore, it is possible to perform mathematical calculations on the relationship between spatial arrangements and objects and backgrounds.

According to the configuration of the present invention as described above, it is possible to search a similar object image quickly, accurately, and efficiently in a large database. Furthermore, an object can be effectively extracted from an image by using an image search framework according to an embodiment of the present invention.

An image retrieval method according to an embodiment of the present invention will be described in more detail below with reference to the accompanying drawings.

FIG. 1 is a flowchart for explaining a process of extracting feature points from a query image and calculating a descriptor.

In step 110, the image search service server may build an image database.

More specifically, the service server includes a crawler, a parser, and an indexer to collect web documents on-line, and access various information linked to images and images included in the web documents.

For example, a crawler might collect a list of web addresses of interest, pass the image-related data to the service server by checking the website and tracking the link. At this time, the parser analyzes the web document collected during the crawling process, extracts the image included in the page, information associated with the image, and the indexer can index the corresponding position and meaning.

Further, the service server can process the collected images. This is for labeling the object based on the image and further providing the search result based on the similarity of the image, without relying on other text information.

To do this, the service server extracts features of the collected images and indexes feature information of the images for efficiency in searching.

More specifically, the service server can detect the feature region of the images (Interest Point Detection). The feature region refers to a main region for extracting a descriptor, that is, a feature descriptor, of an image for judging whether or not there is a similarity among images.

According to the embodiment of the present invention, such a feature region is a region including a corner, a corner or the like out of the contour including the image, a blob distinguished from the surrounding region, a region which is unchanged or crooked according to the deformation of the image, It can be a dark or bright polar point and can be a patch (piece) of an image or an entire image.

Furthermore, the service server can extract the descriptor from the feature region (Descriptor Extraction). The feature descriptor is a vector value representing the characteristics of the image.

According to an embodiment of the present invention, the feature descriptor may be calculated using the position of the feature region for the image, or the brightness, color, sharpness, gradient, scale, or pattern information of the feature region. For example, the feature descriptor may be calculated by converting the brightness value, the brightness value, or the distribution value of the feature region into a vector.

According to the embodiment of the present invention, the feature descriptor for the image may include not only a local descriptor based on the feature region, but also a global descriptor, a frequency descriptor, a binary descriptor, It can be expressed as a Neural Network descriptor.

More specifically, the feature descriptor may include a global descriptor (e.g., a global descriptor) for converting the brightness, color, sharpness, gradient, scale, pattern information, etc. of each of the regions divided by the entire image or an arbitrary reference, Global descriptor).

For example, the feature descriptor may include a frequency descriptor for converting the number of times the specific descriptors classified into the image are included in the image, the number of inclusions of the global feature such as the previously defined color table, etc. into a vector value, A binary descriptor which extracts whether the size of each element value constituting the descriptor is larger or smaller than a specific value and then transforms it into an integer type and uses it in a layer of a neural network Or a neural network descriptor for extracting image information used for classification.

Further, according to an embodiment of the present invention, the service server can apply a VT (Vocabulary Tree) for encoding the descriptor extracted from the image to the first level and the second level separately.

The first level VT can be formed by constructing a rough code book from the image to be searched by considering the values of the image descriptor as the concept of words in Bow (Back of Words). Further, according to an embodiment of the present invention, the service server applies the first level VT to the collected image, composes the descriptors created with the first level VT into one vector, constructs the low dimensional vector, can do.

Furthermore, the service server according to the embodiment of the present invention creates a second level VT for the database image in a manner of constructing a refined codebook that is more refined by reflecting the statistical characteristics of the geometry in the image local area, It can be created with the metadata of the image and indexed into that image.

A more detailed description of generating the first level VT and the second level VT will be described later in the description of FIGS. 2 and 3 attached hereto.

In step 120, the service server may receive a query to query the client for an image search.

To do this, the client can run a program to generate queries and receive results. The program may be an extension program installed in a mobile terminal or a web browsing tool of a desktop in the form of a plug-in, an active X, or the like, or may be an application installed in a mobile terminal.

The program can be downloaded and installed from a service server according to an embodiment of the present invention, and further downloaded from a service server through a website server such as an online application market.

When the service server receives the query, it can identify the image included in the query and pre-process the image for later retrieval. The pre-processing of the query image, unlike the subsequent image processing, is to change the scale of the query image itself for retrieval according to a predetermined format, or to change the size or resolution.

Further, in step 130, the service server may extract feature points from the query image. The service server according to an embodiment of the present invention includes a blob that is divided into corners such as a corner and a surrounding area among contours and contours included in an image in order to select a location having invariance in image transformation in the query image, , A region that is unchanged or crooked according to the deformation of the image, or a pole having a darker or lighter characteristic than the surrounding brightness, and can extract a patch (image) of the image or the entire image.

Further, in step 140, the service server may extract the descriptor from the feature region of the query image (Descriptor Extraction). The feature descriptor is a vector value representing the characteristics of the image.

According to an embodiment of the present invention, the feature descriptor may be calculated using the position of the feature region for the image, or the brightness, color, sharpness, gradient, scale, or pattern information of the feature region. For example, the feature descriptor may be calculated by converting the brightness value, the brightness value, or the distribution value of the feature region into a vector.

According to the embodiment of the present invention, the feature descriptor for the image may include not only a local descriptor based on the feature region, but also a global descriptor, a frequency descriptor, a binary descriptor, It can be expressed as a Neural Network descriptor.

More specifically, the feature descriptor may include a global descriptor (e.g., a global descriptor) for converting the brightness, color, sharpness, gradient, scale, pattern information, etc. of each of the regions divided by the entire image or an arbitrary reference, Global descriptor).

For example, the feature descriptor may include a frequency descriptor for converting the number of times the specific descriptors classified into the image are included in the image, the number of inclusions of the global feature such as the previously defined color table, etc. into a vector value, A binary descriptor which extracts whether the size of each element value constituting the descriptor is larger or smaller than a specific value and then transforms it into an integer type and uses it in a layer of a neural network Or a neural network descriptor for extracting image information used for classification.

FIG. 2 is a flowchart illustrating a process of extracting a candidate group of a database image similar to a query image according to an embodiment of the present invention.

In FIG. 2, the service server can extract a candidate list of a database image similar to a query image, and this can be performed using a database image extracted in FIG. 1 and a descriptor of a query image. (Step 210)

In step 220, the service server may consider the values of the descriptors of the query image as a concept of words in Bow (Back of Words) and construct a rough codebook. In the present invention, this is referred to as a first level VT.

Further, the service server may integrate the local features included in the first level VT into one to form a low dimensional vector and index it. (Steps 220-240)

The index creation criterion is applied to the database image at step 110 of FIG. 1, and an index for the first level VT is formed in each of the images in the database.

Therefore, the service server can form the similar image candidate group by comparing the index value of the query image with the index value of the database image and regarding the images having the distance value less than the predetermined range as similar. (Step 250)

3 is a flowchart illustrating a process of setting a ranking of an image candidate group by applying a topic model according to an embodiment of the present invention.

In FIG. 2, the service server can extract a candidate list of a database image similar to a query image, and this can be performed using a database image extracted in FIG. 1 and a descriptor of a query image. (Step 310)

More specifically, in order to set ranking of the similar image list extracted in FIG. 2, the service server configures the values of the descriptors of the query image into a refined codebook in which the statistical characteristics of the geometry in the local area are refined, can do. In the present invention, this is referred to as a second level VT. (Step 310)

Furthermore, the service server may apply the topic model to the second level codebook of the query image and the second level codebook of the candidate list image generated in Fig. (Step 320)

The topic model is for estimating the model verification probability by repeatedly verifying whether or not the model is applied, assuming a model that distinguishes the background from the object in the image. The topic model can be learned by a machine learning method of extracting a morphological distribution and background distribution from a large number of images and distinguishing objects of the image.

When the topic model is applied, the subject area and the background area are distinguished from each other in the query image and the candidate list image, and the similarity between the query image topic object and the topic object of the candidate list image can be compared. (Steps 330 and 340)

A topic model according to an embodiment of the present invention will be described in more detail with reference to Figs. 5 and 6 attached hereto.

5 is a diagram for explaining an example of applying a topic model in an image according to an embodiment of the present invention.

5 compares the GV (Geometric Verification) of the reference image with the comparison image.

In Figure 5A, the global transformation of GV reflects the transformation of the "building" because the property match of the building dominates the functional conformity of the other entity. This correspondence includes other matching entities (airplanes, automobiles in the example of FIG. 5A), but the matchpoints indicated in blue are considered to be matched, but can be treated as not matching the matchpoints indicated in yellow.

5B is an illustration of estimating local geometric consistency applying a topic label according to an embodiment of the present invention. In the example of FIG. 5B, the topic label may first be assigned to a feature associated with the same object. According to the embodiment of the present invention, since the topic label is first assigned to the object, and then the geometric consistency of each local subject area is estimated, unlike FIG. 5A, the matching points of the objects can be processed as matching.

FIG. 6 is a diagram for explaining a relationship between parameters and parameters applied to an SLTMB (Spatial Latent Topic Model with Background Distribution) according to an embodiment of the present invention.

A topic model according to an embodiment of the present invention is for estimating the background of the variance and potential topics of the space, and can be learned by a machine learning method.

The parameters and parameters shown in FIG. 6 can be described with reference to Table 1 below.

Further, a topic model according to an embodiment of the present invention may be updated through the following steps.

Further, a topic model according to an embodiment of the present invention may be updated through the following steps.

For each topic t, the morphological distribution Φt ~ Dir (β1) and the background distribution Ω ~ Dir (β2)

(T = 1, ..., D) for each image Id (d = 1 ... D) 2, ..., T), the position distribution ~ NW (μ0, κ, ν, Q) can be calculated.

Further, switch sample s _di ~ function (λ _d ) can be calculated for each word position value iε {1,2, ..., N _d }. Further, it is possible to calculate the topic z _di ~ function (? _D ) at s _di = 1, calculate the visual word w _di ~ function (? _Zdi ), and calculate the word position value l _di . Next, we can calculate the visual word w _di ~ function (Ω) at s _di = 2 and calculate the word position value l _di ~.

Further, a topic model according to an embodiment of the present invention can be calculated according to the following equation (1).

Referring again to FIG. 3, the service server can apply the topic model to set the ranking of the candidate image group based on the similarity degree between the candidate image group and the query image. (Step 350)

At this time, the similarity calculation according to the embodiment of the present invention can be calculated using the following Equations (2) to (6).

The service server may then send the image search results to the client. (Step 360)

4 is a diagram for explaining a configuration of an image retrieval framework according to an embodiment of the present invention.

An image search framework in accordance with an embodiment of the present invention may include a preprocessed image database 460.

Further, upon receiving the query image 410, the image list 480 can be presented in a similar image order as compared with the database image 460.

More specifically, the image retrieval framework according to an embodiment of the present invention extracts a feature (descriptor 420 for a feature point) upon receiving a query image and processes it in two steps.

First, the image search framework can create a two-level VT for the query image.

The first codebook may be roughly created to form a low dimensional vector 447 integrated (445) for the local feature. Which can be indexed for the query image in the indexing module 465 and used to extract the candidate set 470 by comparing it with the index value of the database image. (Encoding phrase, 440)

The second codebook refers to a refined codebook that is more refined by reflecting the statistical properties of the geometry in the local area of the image. This can be used to generate a second level VT 455 and apply the topic model 457 to the second level VT to calculate 475 the similarity of the candidate image list to the query image. (Topic Modeling Phrase, 450)

The image retrieval framework according to the embodiment of the present invention may present the retrieval result 480 in the order of high similarity of images through the encoding step and the topic modeling step.

7 is a diagram for explaining an example in which search results are displayed in a client according to another embodiment of the present invention.

In particular, the embodiment of FIG. 7 relates to a method capable of improving the speed of the entire system by taking charge of processing by the client to generate feature information of the query image, thereby reducing the amount of data moved to the network

The contents of step 710, step 720, step 755, and step 760 of FIG. 7 are substantially the same as those of FIGS. 1 to 6 described above.

In step 725, the client can proceed to distinguish objects included in the displayed image. For example, a client can distinguish objects such as a background, a person, a coffee can, and a lotion using the outline of the displayed image.

In step 730, the client may receive a user selection input for the delimited image object. At this time, the client can set the object such as the background or the person to be selected by the user according to the purpose of the implementation, and to select only the object for the merchandise available for sale.

In step 740, the client may perform a process of generating feature information on the object image selected by the user as a query image. More specifically, the client can detect the feature region of the image object selected by the user and extract the feature descriptor. The concrete contents are the same as those described in the description of FIG. 1 to FIG. On the other hand, the number of characteristic regions and the number of characteristic descriptors in the query image can be set differently for each client considering the performance of the process.

Thereafter, in step 745, the client can transmit the feature information of the generated query image to the server.

According to the embodiment of FIG. 7, not only the image displayed on the client but only the feature information of the object image selected by the user can be moved through the network. Therefore, the amount of data moved to the network can be greatly reduced, and the time required for uploading and downloading is reduced, thereby improving the speed of the entire system.

On the other hand, in step 765, the service server can perform the process of converting the received query image feature information according to the same criterion as that of classifying the feature information of the database image.

The contents of steps 790 and 740 are substantially the same as those described with reference to Figs. At step 7750, the client may display the search results.

 8 is a block diagram illustrating a configuration of a client according to an embodiment of the present invention. Although a mobile terminal such as a smart phone, a tablet, and a PDA is described in this specification, the present invention is not limited thereto, and a client according to an embodiment of the present invention may be a personal computer or a notebook computer.

8, the client 800 according to the embodiment of the present invention includes a communication unit 810, a touch screen unit 830, an input unit 820, a storage unit 840, and a control unit 860 And may further include an audio processing unit and a video processing unit, though not shown in FIG.

The communication unit 810 performs a function of transmitting and receiving corresponding data for wired / wireless communication of the client 800. The communication unit 810 receives data through a wired / wireless channel and outputs the received data to the control unit 850, And transmits the data through the channel.

The touch screen unit 830 includes a touch panel 833 and a display unit 836. Particularly, the touch panel 833 according to the embodiment of the present invention can receive the user's selection input of the goods object among the images displayed on the display unit 836, and the display unit 836 can display the goods information .

The input unit 820 receives an input of a user for controlling the client 800, generates an input signal, and transmits the input signal to the controller 850. In the preferred embodiment of the present invention, the client 800 is fully operable only with the touch screen portion 830. [ In this case, the touch panel 833 can perform the function of the input unit 820.

The storage unit 840 stores programs and data necessary for the operation of the client 800, and can be divided into a program area and a data area. The data area of the storage unit 840 according to an embodiment of the present invention may store feature information of a query image or a query image. Further, the program area of the storage unit 840 according to the embodiment of the present invention may store an application for causing the client to execute the process of displaying the product information. The application may include a function of transmitting an image displayed on the display unit 836 to a server as a query image, a function of receiving product information identical or similar to the query image from the server, a function of displaying the product information Can be performed.

The control unit 850 controls the overall operation of each component of the client. In particular, the control unit 850 of the client according to the embodiment of the present invention transmits the image displayed on the display unit 836 to the server as a query image, and transmits a search result, which is the same as or similar to the query image, 810, and controls the display unit 836 to display the search result.

9 is a block diagram illustrating a configuration of a server 900 according to an embodiment of the present invention. 9, the server 900 according to the embodiment of the present invention may include a communication unit 910, a storage unit 930, and a control unit 920, and may include an input unit And may further include a display unit.

9, the server 900 is illustrated as including a communication unit 910, a storage unit 930, and a control unit 920, but each block may be physically separated. For example, the storage unit 930 physically exists in a database (not shown) and may be connected to the controller 920 of the server 900 through the communication unit 910.

The communication unit 910 performs a function of transmitting and receiving data for wired / wireless communication of the server 900. The communication unit 910 receives data through a wired / wireless channel and outputs the data to the control unit 920. The control unit 920 transmits the data output from the control unit 920 to a non- Through the network. In particular, the communication unit 910 according to the embodiment of the present invention can perform a function of receiving a query image from a client.

The storage unit 930 stores programs and data necessary for the operation of the server 900, and can be divided into a program area and a data area. In the data area of the storage unit 930 according to an embodiment of the present invention, the storage unit 930 may store an image database, text information associated with the image, and query image feature information.

Further, the program area of the storage unit 940 according to the embodiment of the present invention may store a computer program that causes the server to execute a process of providing an image search result to a client. The computer program includes a function of generating a database for a plurality of images and extracting feature information of an image included in the database, a function of receiving from the client an image displayed on the client as a query image, A function of determining a matching image matching the feature information of the query image by searching the feature information of the database image, and providing the client with information about the matching image.

The control unit 920 controls the overall operation of each component of the server 900. Particularly, the controller 920 according to the embodiment of the present invention generates a database for a plurality of images, extracts feature information of the images included in the database, searches the feature information of the database image, It is possible to control the communication unit 910 to determine the matching image with the feature information of the query image and provide the client with information about the search result

The embodiments of the present invention disclosed in the present specification and drawings are merely illustrative of specific embodiments of the present invention and are not intended to limit the scope of the present invention in order to facilitate understanding of the present invention. It is to be understood by those skilled in the art that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein.

800: Client
810:
820: Input section of the client
830: Client's touch screen
840: Storage portion of the client
850:
800: Server
810:
820:
830: Storage section of the server

Claims (13)

In a service server, a method for performing an image search,
Generating a first descriptor vector by applying a first level codebook in the query image and extracting at least one candidate image similar to the query image by comparing the query image with a database image;
Extracting a second descriptor vector by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor vector, A step B of estimating a subject area of the subject;
C) estimating a subject area of the candidate image by applying the topic model to the candidate image; And
Calculating a degree of similarity between a subject area of the query image and a subject area of the candidate image; and setting a similarity ranking of the candidate image according to the degree of similarity. The method according to claim 1,
Collecting a plurality of images and applying the first level codebook to the collected images to generate a first descriptor vector;
Generating a second descriptor vector by applying the second level codebook to the collected image;
And indexing the first descriptor vector and / or the second descriptor vector into an image using an index value. 3. The method according to claim 2,
And estimating a subject area of the candidate image by applying the topic model to the second descriptor vector of the candidate image. 4. The method according to claim 3, wherein in step (D)
And estimating a similarity between the candidate image and the query image by applying a weight to a descriptor of a subject area of the candidate image. In a service server, a method for performing an image search,
Collecting a plurality of images, extracting a first descriptor by applying a first level code book to the collected images, and constructing a database indexing the vectors of the first descriptors into corresponding images as index values;
And generating a first descriptor vector by applying the first level codebook in the query image, comparing the vector with an index value of the database image to extract a candidate image similar to the query image, ;
Extracting a second descriptor by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor, Estimating a region; And
And estimating a subject area of the candidate image by applying the topic model to a second descriptor extracted by applying the second level codebook to the candidate image. 6. The method of claim 5,
Calculating a degree of similarity between a subject area of the query image and a subject area of the candidate image, and setting a similarity ranking of the candidate image according to the degree of similarity. An apparatus for performing an image search,
A database comprising at least one image;
A communication unit for receiving a query image;
A descriptor extracting unit for generating a first descriptor vector by applying a first level codebook in the query image and extracting a second descriptor vector by applying a second level codebook having a higher dimension than the first level codebook;
Estimating a subject area of the query image by applying a topic model for distinguishing a subject area and a background area to the second descriptor vector of the query image, applying the topic model to the candidate image, Estimating a subject area estimator; And
Extracting at least one candidate image that is similar to the query image as compared with the first descriptor vector and the database, calculating the similarity between the subject region of the query image and the subject region of the candidate image, And a degree-of-similarity calculation unit that sets a degree of similarity rank of the image. 8. The method of claim 7,
Wherein the descriptor-
Generating a first descriptor vector by applying the first level codebook to an image included in the database and applying a second level codebook to generate a second descriptor vector, 9. The method of claim 8,
And an index unit for indexing the first descriptor vector and / or the second descriptor vector into an image using an index value. An apparatus for performing an image search,
An image database for extracting a first descriptor by applying a first level codebook in a plurality of images and indexing a vector for the first descriptor into an image using an index value;
A communication unit for receiving a query image; And
Generating a first descriptor vector by applying the first level codebook in the query image, extracting at least one candidate image similar to the query image by comparing the vector with an index value of the database image, Extracts a second descriptor by applying a second level codebook of a higher level than the first level codebook and estimates a subject area of the query image by applying a topic model for distinguishing the subject area and the background area to the second descriptor And a controller for applying the topic model to a second descriptor extracted by applying the second level codebook to the candidate image to estimate a subject area of the candidate image. 11. The method of claim 10,
Wherein,
Wherein the similarity degree calculating unit calculates a degree of similarity between a subject area of the query image and a subject area of the candidate image and sets a similarity ranking of the candidate image according to the degree of similarity. A computer program recorded on a recording medium for causing a computer to execute a method of retrieving an image in a service server,
A function of collecting a plurality of images;
A function of building a database of the collected images;
A first level descriptor is applied to the query image to extract a first descriptor, a vector for a first level descriptor is generated, and an index value of the database image is compared with the vector, A function of extracting at least one candidate image similar to the image;
Extracting a second descriptor by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor, A function of estimating a region;
Estimating a subject area of the candidate image by applying the topic model to the second descriptor of the candidate image; And
Calculating a degree of similarity between a subject area of the query image and a subject area of the candidate image, and setting a similarity ranking of the candidate image according to the degree of similarity. A computer program recorded on a recording medium for causing a computer to execute a method of retrieving an image in a service server,
The ability to create an image database;
Generating a first descriptor vector by applying a first level codebook in the query image and extracting at least one candidate image similar to the query image by comparing with the database image;
Extracting a second descriptor vector by applying a second level codebook having a higher dimension than the first level codebook in the query image and applying a topic model for distinguishing a subject area and a background area to the second descriptor vector, A function of estimating a subject area of the subject;
Applying the topic model to the candidate image to estimate a subject area of the candidate image; And
Calculating a degree of similarity between a subject area of the query image and a subject area of the candidate image, and setting a similarity ranking of the candidate image according to the degree of similarity.

Images