KR101768521B1 - Method and system providing informational data of object included in image - Google Patents

Abstract

Disclosed are a method and a system for providing data on an object included in an image. The method for providing data on an object included in an image comprises the steps of: receiving an image by using a query for retrieval; detecting an object corresponding to a specific item in the image by using a deep learning model in which the specific item is learned and extracting features of the detected object; retrieving data corresponding to the features of the object among data related to the specific item; and providing the data corresponding to the features of the object as a search result for the image.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and system for providing information data on an object included in an image,

The following description relates to techniques for finding and providing associated information data for objects included in an image.

Various kinds of information providing windows for displaying various information data such as advertisements are provided on various kinds of portal, professional or personal web sites on the Internet through online.

Information data through the Internet is forming a huge profitable market in terms of function and effectiveness in addition to media such as TV, newspaper, and radio. In addition, portals and websites providing various information have a larger share of imports from information data.

Information data to be displayed on a web site of the Internet is provided in various forms such as a banner or a moving image or a user can visually recognize. Such information data can be simply displayed on an information providing window designated on a main page or a sub page of a web site, or can be divided into search related products for information data on a corresponding search word.

As an example of a technology for providing information data, Korean Patent Laid-Open No. 10-2010-0004312 entitled " Method of Interacting Web Site Advertisements Through the Internet ", an advertiser concludes an advertisement with an arbitrary Web site, A plurality of websites linked with sites are linked with each other, and a desired advertisement is simultaneously linked through a desired web site.

There is provided an information providing method and system capable of detecting an object included in an image by targeting all images without constraint and searching for information data related to the detected object.

The present invention provides a method and system for providing information for finding an object corresponding to an item of a specific topic in an image using an object detection algorithm based on CNN (Convolutional Neural Network features).

There is provided an information providing method and system capable of classifying a category of an object detected in an image and extracting characteristics of a specific attribute by applying a hierarchical classification model and an attribute learning based CNN.

The present invention provides a method and system for providing information that can more accurately and quickly find related information in a large-capacity DB through indexing and searching based on a sparse characteristic of an object detected in an image.

There is provided an information providing method and system for providing a search result through a dimension reduction and a quantization process for a CNN-based deep feature having a high dimension.

A computer-implemented method, comprising: receiving an image in a query for search; Extracting a feature of the detected object after detecting an object corresponding to the specific item in the image using a deep learning learning model learned for a specific item; Retrieving information data corresponding to a characteristic of the object as information data related to the object among information data related to the specific item; And providing information data corresponding to the feature of the object as a search result for the image.

1. A computer-implemented system, comprising: an image receiving unit for receiving an image as a query for a search; An object detecting unit for detecting an object corresponding to the specific item in the image using a deep learning learning model learned for a specific item and extracting features of the detected object; An information retrieval unit for retrieving information data corresponding to a feature of the object as information data related to the object among information data related to the specific item; And an information providing unit for providing information data corresponding to the feature of the object as a search result for the image.

By using object detection algorithm based on CNN (Convolutional Neural Network features), it is possible to accurately find an object corresponding to an item of a specific subject in an image without any constraint condition, The result can be improved.

The hierarchical classification model and CNN based on attribute learning can be applied to classify the categories of the objects detected in the images and to extract the characteristics of specific attributes.

Based on the sparse characteristics of objects detected in the image, high-speed indexing and searching can be used to find relevant information more accurately and quickly in a large-capacity DB.

The search quality can be improved by applying dimensionality reduction and quantization to the ranking of related information for a CNN-based deep feature having a high dimension.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention.
3 is a diagram showing an example of an annotation tool for constructing a data set in an embodiment of the present invention.
4 is a diagram illustrating an example of a component that a processor of a server according to an embodiment of the present invention may include.
5 is a flowchart illustrating an example of a method that a server according to an embodiment of the present invention can perform.
FIG. 6 is a diagram showing an example of a process of regenerating a pre-learning model for category classification, according to an embodiment of the present invention.
FIG. 7 is a diagram showing examples of localization and attribute learning classification results for an image in an embodiment of the present invention.
FIG. 8 is a diagram illustrating an example of a CNN-based image characteristic classification process according to an exemplary embodiment of the present invention.
9 is a diagram for explaining an image characteristic having a rare characteristic in an embodiment of the present invention.
10 is a diagram showing an example of an indexing process using a rare property of an image characteristic in an embodiment of the present invention.
11 is a diagram showing an example of a dimension reduction and quantization process in an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. 1 shows an example in which a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170 are included. 1, the number of electronic devices and the number of servers are not limited to those shown in FIG.

The plurality of electronic devices 110, 120, 130, 140 may be a fixed terminal implemented as a computer device or a mobile terminal. Examples of the plurality of electronic devices 110, 120, 130 and 140 include a smart phone, a mobile phone, a navigation device, a computer, a notebook, a digital broadcast terminal, a PDA (Personal Digital Assistants) ), And tablet PCs. For example, the electronic device 1 110 may communicate with other electronic devices 120, 130, 140 and / or the servers 150, 160 via the network 170 using a wireless or wired communication scheme.

The communication method is not limited, and may include a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 170 may include, as well as a short-range wireless communication between the devices. For example, the network 170 may be a personal area network (LAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN) , A network such as the Internet, and the like. The network 170 may also include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, It is not limited.

Each of the servers 150 and 160 is a computer device or a plurality of computers that communicate with a plurality of electronic devices 110, 120, 130 and 140 through a network 170 to provide commands, codes, files, Lt; / RTI > devices.

In one example, the server 160 may provide a file for installation of the application to the electronic device 1 (110) connected via the network 170. [ In this case, the electronic device 1 (110) can install an application using a file provided from the server (160). The server 150 is connected to the server 150 in accordance with an operating system (OS) included in the electronic device 1 110 or under control of at least one program (for example, a browser or an installed application) I can receive contents. For example, when the electronic device 1 (110) transmits a service request message to the server 150 via the network 170 under the control of the application, the server 150 transmits a code corresponding to the service request message to the electronic device 1 The first electronic device 110 can provide contents to the user by displaying and displaying a screen according to the code according to the control of the application.

In particular, the server 150 receives an image as a query, finds an object included in the image, and searches the information data related to the object to provide the search result to the electronic device 1 (110). In one example, the server 150 may receive an image query from the electronic device 1 110 via the network 170, retrieve information data matching or similar to the objects contained in the image, 1 (110). As another example, the server 150 can recognize an image included in an image or a document requested by the electronic device 1 (110) as a query, retrieve information data matching or similar to the object included in the image, (110) with the image or document requested for access by the electronic device (110).

2 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention. In FIG. 2, an internal configuration of the electronic device 1 (110) as an example of one electronic device and the server 150 as an example of one server will be described. Other electronic devices 120, 130, 140 or server 160 may have the same or similar internal configurations.

The electronic device 1 110 and the server 150 may include memories 211 and 221, processors 212 and 222, communication modules 213 and 223 and input / output interfaces 214 and 224. The memories 211 and 221 may be a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. The memory 211 or 221 may store an operating system or at least one program code (for example, a browser installed in the electronic device 1 (110) and a code for an application for video communication). These software components may be loaded from a computer readable recording medium separate from the memories 211 and 221. [ Such a computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, and a memory card. In other embodiments, the software components may be loaded into memory 211, 221 via communication modules 213, 223 rather than a computer readable recording medium. For example, at least one program may be a program installed by a file distribution system (for example, the server 160 described above) that distributes installation files of developers or applications, May be loaded into the memory 211, 221 based on the application described above.

Processors 212 and 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processors 212 and 222 by the memories 211 and 221 or the communication modules 213 and 223. For example, the processor 212, 222 may be configured to execute a command received in accordance with a program code stored in a recording device, such as the memory 211, 221.

The communication modules 213 and 223 may provide functions for the electronic device 1 110 and the server 150 to communicate with each other through the network 170 and may provide functions for communicating with other electronic devices (for example, the electronic device 2 120) Or to communicate with another server (e.g., server 160). For example, when the processor 212 of the electronic device 1 110 receives a request (for example, a request for a video call service) generated according to a program code stored in a recording device such as the memory 211, To the server 150 via the network 170 in accordance with the < / RTI > Conversely, control signals, commands, contents, files, and the like provided under the control of the processor 222 of the server 150 are transmitted to the communication module 223 of the electronic device 110 via the communication module 223 and the network 170 213 to the electronic device 1 (110). For example, control signals and commands of the server 150 received through the communication module 213 may be transmitted to the processor 212 or the memory 211, May be stored as a storage medium that may further include a < RTI ID = 0.0 >

The input / output interfaces 214 and 224 may be means for interfacing with the input / output device 215. For example, the input device may include a device such as a keyboard or a mouse, and the output device may include a device such as a display for displaying a communication session of the application. As another example, the input / output interface 214 may be a means for interfacing with a device having integrated functions for input and output, such as a touch screen. More specifically, the processor 212 of the electronic device 1 (110) uses the data provided by the server 150 or the electronic device 2 (120) in processing commands of the computer program loaded in the memory 211 A service screen or contents can be displayed on the display through the input / output interface 214. [

Also, in other embodiments, electronic device 1 110 and server 150 may include more components than the components of FIG. However, there is no need to clearly illustrate most prior art components. For example, electronic device 1 110 may be implemented to include at least a portion of input / output devices 215 described above, or may be implemented with other components such as a transceiver, Global Positioning System (GPS) module, camera, Elements. More specifically, when the electronic device 1 (110) is a smart phone, it may be an acceleration sensor, a gyro sensor, a camera, various physical buttons, buttons using a touch panel, input / output ports, It is to be understood that the present invention may be embodied in various forms without departing from the spirit or scope of the invention.

Embodiments of the present invention are directed to techniques for retrieving and recommending information data associated with an image by querying a natural image without constraints based on a deep learning method.

In the present invention, the object included in the image is detected and classified (localized), and information data related to the detected object can be quickly and accurately searched in the large-capacity DB, and the search result can be provided in the order matching or similar to the object.

To this end, in the present invention, a CNN-based localization method may be applied to an image and a fine-tuning method may be provided for a specific subject (e.g., fashion, etc.) in learning. The present invention can be applied to category classification and feature extraction by applying a hierarchical classification model and CNN based on attribute learning for excellent searching. Basically, data on the Internet (e.g., shopping-related data, etc.) is vast. For this reason, it is very important to do a quick search, which can provide a very simple and robust fast indexing and search method with a rare concept.

In this specification, an image is included in a document such as a web page, an image captured or generated on a mobile terminal or other type of electronic device, or an image in a form distributed or displayed on the Internet (e.g., shopping mall, SNS, The image may include all kinds of images such as an image.

In the following embodiments, an image including a fashion item is described as a representative example, but the present invention is not limited thereto.

For an image-based fashion search, first, the subject must be precisely searched regardless of the complex background in the image. Second, accurate image classification should be done to determine exactly which category the object found belongs to. Third, you need to extract more robust image features to ensure that similar products appear as search results. Fourth, items such as fashion have a huge DB, so you have to find them faster and more accurately.

First, the server 150 may configure a data set for image-based fashion retrieval in advance, and the data set configuration principle is to display the characteristics of the fashion category well. For this purpose, the data set can be constructed by dividing the attribute of fashion into three attributes, which are largely representative, and dividing it into detailed attributes. These attribute-based datasets can be applied to deep-learning learning, and in particular Convolutional Neural Network features (CNN), an area of deep-learning, can be applied. For example, to learn the CNN model, it is possible to acquire images including fashion items such as clothes wearing clothes through various websites such as online shopping mall, web engine, and SNS. And, a data set for detecting fashion items (objects), a data set for localization learning and a data set for property learning can be configured. For example, data sets for localization learning can be collected by dividing into clothes, shoes, bags, etc., and this form is for applying CNN-based object detection algorithm. At this time, each of the fashion items included in the image has one label and corresponding ROI (region of interest). To configure such a dataset, the server 150 may provide a web-based annotation tool. For example, as shown in FIG. 3, the server 150 can provide various types of annotation-capable tools for the fashion items 301 included in the image 300, To allow multiple people to create a dataset at the same time. Next, the data set for property learning can be applied to the ROI detected in the image. In the present invention, the data set for attribute learning can be configured based on various colors, textures, and category attributes that can express fashion attributes well. The category attribute can be basically composed of detailed categories for each upper level by hierarchical structure. The category classified in localization means upper category, and can be divided into each detailed category of fashion based on this.

Based on the data set reflecting the characteristics of the fashion category as described above, the server 150 can accurately find the object corresponding to the fashion item in the image, and can identify exactly which color, texture, and category Can be accurately classified.

Hereinafter, a specific embodiment of a system and method for providing an image-based fashion search result will be described.

FIG. 4 is a diagram illustrating an example of a component that a server of a server according to an embodiment of the present invention can include; FIG. 5 is a diagram illustrating an example of a method that a server can perform according to an exemplary embodiment of the present invention; Fig.

4, the processor 222 of the server 150 may include an image receiving unit 410, an object detecting unit 420, an information searching unit 430, and an information providing unit 440 have. The processor 222 and the components of the processor 222 may control the server 150 to perform the steps S510 through S540 included in the method of FIG. At this time, the components of the processor 222 and the processor 222 may be implemented to execute instructions according to the code of the operating system and the code of at least one program that the memory 221 contains. In addition, components of processor 222 may be representations of different functions performed by processor 222 in accordance with control commands provided by the operating system or by at least one program. For example, the image receiving unit 410 may be used as a functional representation in which the processor 222 receives an image according to the control command described above.

In step S510, the image receiving unit 410 may receive the image as a query for searching. At this time, the image receiving unit 410 may receive an image including an object corresponding to the fashion item as an input. For example, the image receiving unit 410 may receive an image directly from the electronic device 1 (110) via the network. As another example, the image receiving unit 410 may receive an image from an internal or external database system. For example, if an image is included in a document or contents to be served to the electronic device 110, the image receiving unit 410 may receive the image from the database system. An image input as a query can mean a natural image without any restrictions. For example, all types of images such as street images, shop images, mobile phone images, and images included in web pages posted in news or SNS can be used as queries.

In operation S520, the object detection unit 420 detects at least one object corresponding to a fashion item in the image received as a query using a CNN learning model to which an item of a specific topic, for example, a data item of a fashion item is applied The feature of the detected object can be extracted.

The object detection unit 420 may perform automatic localization of an object included in the image so that the user does not require any interaction for selecting a fashion item existing in the image. In the present invention, the object detection unit 420 applies a localization method based on a CNN-based object detection algorithm. At this time, the CNN-based object detection algorithm can detect objects to be classified in an image by learning based on fine- . For example, the object detecting unit 420 may use an R-CNN (Regions with Convolutional Neural Network Features) -based classification model to find an object corresponding to an item of a specific topic in an image. The pre-trained model for object detection uses the CNN learning model learned by using the data set related to the image, and regenerates the classification model for the fine-tuning which is more suitable for fashion in the CNN learning model can do.

Localization in the object detection unit 420 means a process of finding a position and an predefined label of an object corresponding to a fashion item to be searched in the image. The object detection unit 420 applies a CNN-based object detection algorithm for localization. Since the object detection unit 420 internally calculates the candidate ROI in the CNN, it can perform the localization more quickly. The pre-training model can be reconfigured to be more suitable for fashion items in order to improve the classification performance of the object. In other words, pre-training models with detailed categories for fine-tuning can be regenerated, and existing pre-training models can be re-trained using fashion data sets. For example, referring to FIG. 6, the initial preliminary training model 601 uses a pre-learned CNN model. In the following process, the preliminary learning model 602 is learned by learning a model using a data set of a fashion category, . Finally, a final model 603 for performing localization can be created by reconfiguring the fashion learning set once again and applying a pre-learning model 602 suitable for the previously generated fashion.

The object detection unit 420 applies a CNN-based object detection algorithm learned by using a data set of a fashion category to an image received as a query, thereby obtaining localization and attribute learning learning can be performed. Figure 7 illustrates an example of localization results for an image. The object detection unit 420 detects the ROIs 701, 702, and 703 of the fashion objects existing in the image 700 based on the CNN-based object detection algorithm when the image 700 is received, Attribute 710 can be categorized. At this time, the position of the object means the result of the localization, and the object attribute 710 can include the results for the color, texture, and category as a result of the attribute learning of the object.

The object detection unit 420 may classify categories by extracting the characteristics of the object through attribute learning for the object detected in the image. The object detection unit 420 can accurately extract and classify the characteristics of the object detected in the image by applying CNN modeled based on specific attributes of the fashion item, i.e., color, texture, and category. Attribute learning is applied to learning by defining various attributes according to a specific fashion category. We can construct a dataset for color, texture, and category attributes, and learn the data with the properties of that property by applying the CNN algorithm. Each color and texture property can generate a common learning model regardless of category. The classification structure for fashion items has a hierarchical structure, which, in terms of category attributes, shares common characteristics among fashion categories. For example, a dress has both attributes of Top and Skirts. As fashion categories are divided into subcategories, there are many cases where they have common characteristics. Therefore, it is necessary to design so as to preserve distinctive characteristics for each category, and to have a hierarchical structure.

The result of classification according to localization means higher level, and category attribute means lower level. 8, when the image 800 is received, the object detecting unit 420 sequentially applies a CNN model (upper level) 801 for localization and a CNN model (lower level) 802 for attribute learning can do. The object detecting unit 420 finds a fashion label and a position (ROI) of the upper concept in the localization process through the CNN model (upper level) 801 when there are objects corresponding to the fashion item in the image 800 . Thereafter, the object detecting unit 420 performs a CNN process in which a category attribute model matched with the label of the object detected in the image 800 and each color / texture attribute model among the attribute learning CNN model (lower level) To obtain three classification results for each object's color / texture / category. (FULL Connected Layer), which is the immediately preceding layer of the CNN soft-max layer (classifier), is classified into a final classification As with the results, three features (color, texture, category) can be extracted for each (Deep Feature).

Accordingly, the object detecting unit 420 can extract an object corresponding to a fashion item from an image and accurately extract the feature of the object by applying a CNN learning model to which a data item of a fashion item is applied.

5, in operation S530, the information searching unit 430 searches for information data related to the object detected in the image, and extracts the characteristic extracted in the step S520 of the information data on the DB Color, texture, category) corresponding to the image data. The information searching unit 430 can search for information data matching or similar to the objects included in the image in the search method using the rare characteristics of the Deep Feature extracted through the CNN learning model in step S520.

In the case of fashion-related data, there is an enormous amount of data in the world, and it is still a steady increase. Therefore, a technique for quickly searching for such a large-capacity DB is essential. Basically, the difficulty of retrieving large DBs is due to the number of DBs and images with high dimensions.

CNN-based image features (Deep Features) are basically high-dimensional features. In the CNN structure, it has a semantic property as it progresses to a higher layer. This means that the feature has a very distinct value for the learned category. In the present invention, a non-linear ReLU (rectification) function can be applied to the hidden layer of the CNN. The ReLU function can contribute to enhance learning speed and accuracy in deep learning and can be expressed as max (0, x). max (0, x) means that positive values are left for the value of the previous layer, and 0 for negative values. A spe- cial feature of this nature is that sparse coding has zero-valued nature of the vector values.

FIG. 9 is a diagram for explaining characteristics of a deep feature having a rare attribute. FIG. a) represents the entire distribution (0,808 of 0000) including 0 in 10000 features, and b) represents the distribution (8058 out of 0000) of only non-zero values. c) represents the distribution for the values for the 4096 vector (most can be found to be zero), and d) represents the scale view for the non-zero portion in c). That is, most binary files (bin) are filled with zeros when the size is 4096. This means that if there is a binary larger than 0, it means a very semantic property or a discriminative value. It can be applied to indexing for retrieval by using the characteristics of such a deep feature. Finally, in order to switch from a high dimensional vector to a word concept, the following equations (1) and (2) can be applied.

In Equation (1), V denotes a vector representing a Deep Feature (color, texture, category) extracted through the CNN learning model, and Sort (V) means that each vector is sorted in descending order.

In Equation (2), a threshold value is applied to a non-zero value by a given value K (Ranking or Value) for the vector values sorted through Equation (1) Is applied to the actual indexing (indexing _(j) ) (index visual word).

10 illustrates an indexing process using rare properties of a deep feature. Referring to FIG. 10, the information searching unit 430 searches for a vector (V) (1) representing a Deep Feature (color, texture, category) S1 extracted through a CNN learning model in descending order (S2). Then, a threshold value is applied to a non-zero value among the sorted vector values by a given value K to generate an index visual word (S3). If a constraint of Ranking K is applied, a word is generated for indexing by K size. Next, the information retrieving unit 430 may retrieve an image corresponding to the word using the word generated in step S3 (S4). In other words, the information retrieving unit 430 generates a word for indexing from a feature extracted from the CNN learning model with respect to the image received in the query, and uses a voting method using inverted indexing It is possible to quickly select candidate images from a large-capacity DB.

Referring again to FIG. 5, in step S540, the information providing unit 440 may provide, as a search result, information data corresponding to the feature of the object detected in the image with respect to the image received in the query. The information data provided as a search result may mean an image corresponding to the feature of the object detected in the image or a content including the image. At this time, the information providing unit 440 may perform re-ranking of the candidate images selected in step S530. In the voting method applying the index visual word, there is a need to improve the result of the voting because the original characteristic value is lost. For example, the information providing unit 440 may perform rearrangement through dimension reduction and quantization with respect to a high-level feature to improve search quality.

11 shows a reduction and quantization process. The information providing unit 440 may reduce the image feature vector for the search using a dimension reduction algorithm (e.g., a Principal Component Analysis (PCA) algorithm or the like). Referring to FIG. 11, a vector representing an image feature may be represented in a coordinate system through a dimensional reduction method such as PCA projection (S5). Since the feature extracted from the learning model is high-order and the number of candidate images increases, the search speed is slowed down. Therefore, the dimension of the image feature can be selected by selecting the number of eigenvalues for the vectors representing the feature . For example, when selecting the number of eigenvalues, it is possible to select the number of the eigenvalues occupying about 80% of the total eigenvalue by the experiment, and the image characteristic can be reduced to the dimension of the selected size. As shown in Fig. 11, the selected eigenvector can be represented by a new coordinate axis (S6), and quantization can proceed based on the axis (S7). Normally, the quantization criterion may mean an average. If the feature value is larger than the average, the quantization can be proceeded by switching to 1, otherwise.

Accordingly, the information providing unit 440 may perform re-ranking of the candidate images selected in step S530 through dimension reduction and quantization with respect to the feature extracted through the CNN learning model, The retrieved results can be provided by applying the reordered rankings.

An example of a service scenario according to the above information providing method is as follows.

1) A user can request a search for the photograph by sending a photograph to the server 150 by taking a photograph (for example, a street fashion photograph, an airport fashion photograph, etc.) in which a fashion item exists using the electronic device 1 110 have.

2) The server 150 can detect an object included in the image by receiving an image requested to be retrieved from the electronic device 1 (110) and performing automatic localization on the received image. At this time, the server 150 can detect an object corresponding to the fashion item among the objects included in the image, using the CNN learning model to which the fine-tuning method suited to the fashion subject is applied.

3) The server 150 extracts and extracts detailed object characteristics of the object in which the object detected in the image belongs, based on the CNN learning model modeled on concrete fashion attributes, and the color and texture of the object in question Can be classified.

4) The server 150 can retrieve the fashion information corresponding to the fashion attribute of the feature extracted in step 3) of the fashion information on the DB. At this time, the server 150 can quickly and accurately retrieve data of a fashion item matching or similar to the object included in the image in the large-capacity DB by using the characteristic of the rare concept of the fashion attribute.

5) The server 150 may provide fashion information found in step 4) to the first electronic device 110 as a search result on the image received in step 1).

Accordingly, the server 150 may analyze the fashion item included in the image for the image requested by the user, and recommend related information of the fashion item matching or similar to the analyzed fashion item to the user.

Other types of service scenarios are:

1) collecting a document (e.g., news related to entertainers) on the web, analyzing images included in the document, and collecting information (fashion information) related to the fashion item (color, texture, category, etc.) . At this time, by collecting the person information (e.g., entertainer name, etc.) included in the document together, the fashion information can be linked with the person information to form a database.

2) When a user accesses a document including an image of a document on the web, it can detect an object included in the image by performing automatic localization on the image. At this time, the server 150 can detect an object corresponding to the fashion item among the objects included in the image, using the CNN learning model to which the fine-tuning method suited to the fashion subject is applied.

3) The server 150 extracts and extracts detailed object characteristics of the object in which the object detected in the image belongs, based on the CNN learning model modeled on concrete fashion attributes, and the color and texture of the object in question Can be classified.

4) The server 150 can retrieve the fashion information corresponding to the fashion attribute of the feature extracted in step 3) of the fashion information on the DB. At this time, the server 150 can quickly and accurately retrieve data of a fashion item matching or similar to the object included in the image in the large-capacity DB by using the characteristic of the rare concept of the fashion attribute. The server 150 can further search for the person information associated with the corresponding fashion item based on the information constructed in the database conversion process.

5) The server 150 exposes information on a fashion item matching or similar to the fashion item of the image included in the document, and exposes the person information associated with the fashion item on the web document accessed by the user .

Accordingly, when there is an image including a fashion item on a document that the user requests access or provides to the user, the server 150 searches for a fashion item matching or similar to the fashion item included in the image, Recommendation information can be provided. At this time, the server 150 may provide information about the wearer of the fashion item as well as the fashion item by additionally providing the person information associated with the searched fashion item.

According to embodiments of the present invention, by using the CNN-based object detection algorithm, it is possible to accurately find an object corresponding to an item of a specific subject in an image for all images without any constraint condition, The result can be improved. According to embodiments of the present invention, the hierarchical classification model and the CNN based on the attribute learning can be applied to classify the category of the object detected in the image and extract the characteristics of the specific property. In addition, according to embodiments of the present invention, it is possible to find related information more accurately and quickly in a large-capacity DB through high-speed indexing and searching based on a rare characteristic of an object detected in an image. Furthermore, according to embodiments of the present invention, a search feature can be improved by applying CNN-based deep features to rankings of related information through dimension reduction and quantization.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit, a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device As shown in FIG. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (21)

In a computer implemented method,
Receiving an image in a query for search;
Extracting a feature of the detected object after detecting an object corresponding to the specific item in the image using a deep learning learning model learned for a specific item;
Retrieving information data corresponding to a characteristic of the object as information data related to the object among information data related to the specific item; And
Providing information data corresponding to a feature of the object as a search result for the image
Lt; / RTI >
Wherein the extracting comprises:
(ROI) and a predefined label for an object corresponding to the specific item, and then, using an attribute model matching the label with respect to the position, Respectively,
Wherein the searching comprises:
Generating a word for indexing of the given value for the feature by applying a threshold for a vector value representing the feature to a given value to select at least a portion of the vector value, Information data is retrieved,
Wherein the providing step comprises:
Selecting a number of eigenvalues for a vector value representing the feature to reduce the dimension for the feature and re-ranking the information data corresponding to the word through the dimension reduction; ≪ / RTI >
Including
Lt; RTI ID = 0.0 > 1, < / RTI > The method according to claim 1,
Wherein the extracting comprises:
Using a Convolutional Neural Network (CNN) learning model to which a data set for localization learning and attribute learning of the specific item is applied
Lt; RTI ID = 0.0 > 1, < / RTI > The method according to claim 1,
Wherein the extracting comprises:
Extracting features of at least one of a color, a texture, and a category of the object
Lt; RTI ID = 0.0 > 1, < / RTI > The method according to claim 1,
Wherein the extracting comprises:
Detecting a location (ROI, region of interest) of the object included in the image through a higher learning model for localization learning; And
Extracting features of at least one of a color, a texture, and a category with respect to the detected position through a sub-learning model for attribute learning;
≪ / RTI > delete delete delete delete The method according to claim 1,
Wherein the receiving comprises:
Receiving a search request for the image with the image from an electronic device,
Wherein the providing step comprises:
Providing the electronic device with information data corresponding to the characteristic of the object
Lt; RTI ID = 0.0 > 1, < / RTI > The method according to claim 1,
Wherein the receiving comprises:
Receiving the image included in the document when the image is included in the document to be provided to the electronic device,
Wherein the providing step comprises:
Providing the information data corresponding to the feature of the object through the document when the document is provided to the electronic device
Lt; RTI ID = 0.0 > 1, < / RTI > The method according to claim 1,
Wherein the providing step comprises:
If the specific item is a fashion item, providing information on a fashion item corresponding to the feature of the object as a search result on the image
Lt; RTI ID = 0.0 > 1, < / RTI > The method according to claim 1,
Collecting information on fashion items from images included in each document with respect to documents on the web, and linking them with the person information included in the document to form a database
Further comprising:
Wherein the providing step comprises:
When the specific item is a fashion item, providing information on the fashion item corresponding to the feature of the object together with the character information as a search result on the image
Lt; RTI ID = 0.0 > 1, < / RTI > In a computer implemented system,
An image receiving unit for receiving an image as a query for searching;
An object detecting unit for detecting an object corresponding to the specific item in the image using a deep learning learning model learned for a specific item and extracting features of the detected object;
An information retrieval unit for retrieving information data corresponding to a feature of the object as information data related to the object among information data related to the specific item; And
An information providing unit for providing information data corresponding to a characteristic of the object as a search result for the image,
Lt; / RTI >
Wherein the object detection unit comprises:
(ROI) and a predefined label for an object corresponding to the specific item, and then, using an attribute model matching the label with respect to the position, Respectively,
The information retrieval unit,
Generating a word for indexing of the given value for the feature by applying a threshold for a vector value representing the feature to a given value to select at least a portion of the vector value, Information data is retrieved,
The information providing unit,
Selecting a number of eigenvalues for a vector value representing the feature to reduce the dimension for the feature and re-ranking the information data corresponding to the word through the dimension reduction; To do
The system comprising: 14. The method of claim 13,
Wherein the object detection unit comprises:
Using a Convolutional Neural Network (CNN) learning model to which a data set for localization learning and attribute learning of the specific item is applied
The system comprising: 14. The method of claim 13,
Wherein the object detection unit comprises:
Extracting features of at least one of a color, a texture, and a category of the object
The system comprising: 14. The method of claim 13,
Wherein the object detection unit comprises:
(ROI) of the object included in the image is detected through a higher learning model for localization learning, and then the ROI of the object is detected through a lower learning model for attribute learning, Extracting at least one feature of color, texture and category for the detected position
The system comprising: delete delete delete delete A computer-readable recording medium storing a program for executing the method according to any one of claims 1 to 4 and 9 to 12.

Images