KR20200020306A - A method for image searching using an peripheral information and an apparatus therefor - Google Patents

Abstract

The present invention relates to a method for image search using ambient information for extracting an image similar to an image search word in a form of an image file by referring to ambient information other than an item to be a search target of information included in the image and to an apparatus thereof. When receiving an image search request including image data from a user device, the apparatus is implemented to extract a search target item and ambient information related to the search target item from the image data and to perform search based on the search target item of the image data in consideration of the ambient information.

Description

A method for image searching using an peripheral information and an apparatus therefor}

The present invention relates to an image retrieval technique, and more particularly, to a method and apparatus for performing an image retrieval by referring to surrounding information other than a search target item included in an image.

Image search can be divided into two types according to the method and purpose of finding an image. The first is to use a text search term to find related images. For example, if you enter the keyword 'flowers', you can see different types of flowers. This type is a traditional search method from the very beginning of the Internet. The second is to enter an image search term, which is an image file, and analyze the image file to find the same or similar image. It's the way computer analysis and internet technology have developed. Compared to the keyword method, the user can intuitively search for an image.

Korean Laid-Open Patent No. 2010-0043823, published on April 29, 2010 (Name: Method for providing an online shopping function using a product image and a shopping mall server)

SUMMARY OF THE INVENTION An object of the present invention is to provide a method and apparatus for image search using surrounding information which extracts an image similar to an image search word in the form of an image file by referring to surrounding information other than an item to be searched among information included in the image. .

In particular, the present invention uses the background image other than the item to be searched, the metadata information of the image, etc. as surrounding information, thereby limiting the search range of the image or modifying the search result, thereby increasing the reliability of the search result or the user. An object of the present invention is to provide a method and apparatus for image retrieval using surrounding information that can further improve satisfaction.

According to an aspect of the present invention, there is provided a method for image retrieval, comprising: receiving, by an apparatus for image retrieval, an image retrieval request including image data from a user device; Extracting a search target item and surrounding information related to the search target item from the image data by the image search request; Performing a search based on a search target item of the image data in consideration of the surrounding information; And providing a search result to the user device.

The extracting may include extracting the remaining image area of the image data except for the search target item as a background image, extracting the peripheral information through analysis of the background image, and extracting metadata of the image data. And extracting the surrounding information from the metadata.

The extracting may include extracting an image region other than the search target item as a background image from the image data, extracting the peripheral information through analysis of the background image, and then referring to metadata of the image data. Modify or supplement the surrounding information, or analyze the metadata of the image data to extract the surrounding information, and analyze a background image of the remaining image area except for the search target item among the image data to obtain the surrounding information. It can be modified or supplemented.

Here, the surrounding information includes one or more of temporal information, spatial information, and person information related to one or more of production, sale, and use of the searched item.

In addition, the performing of the search may include: specifying a search range by analyzing the surrounding information; Performing an image search based on the search target item; And modifying a search result of the image search based on the specific search range or specifying a search range by analyzing the surrounding information. And performing an image search based on the search target item within the search range.

In addition, the present invention provides an apparatus for image retrieval as another means for solving the above-mentioned problems. An apparatus for searching for an image according to the present invention may include: an image processing module configured to select image to be searched among a plurality of searchable objects included in the image data by analyzing image data requested for image searching; A peripheral information processing module for extracting peripheral information related to the search target item by analyzing at least one of a background image excluding the search target item among the image data and metadata of the image data; And an image search module configured to search for an image similar to the image data based on the search target item in consideration of the surrounding information.

The peripheral information processing module extracts the peripheral information through analysis of the background image among the image data, and then corrects or supplements the peripheral information with reference to the metadata of the image data, or analyzes the metadata of the image data. The peripheral information may be extracted, and the peripheral information may be corrected or supplemented by analyzing the background image.

The image search module analyzes the surrounding information to specify a search range, performs an image search based on the search target item, and then corrects a search result of the image search based on the specific search range, or A search range may be specified by analyzing surrounding information, and an image search may be performed based on the search target item within the specific search range.

According to the present invention, in performing an image search, in addition to the item to be searched, additional information related to items such as temporal information and spatial information is extracted from the background image and metadata of the image, and the search range is limited based on the extracted information. By performing a search or adjusting the ranking of the search results, the user's trust and satisfaction with the image search results provided to the user can be further shaped.

1 is a diagram illustrating an image retrieval system according to an exemplary embodiment of the present invention.
2 is a block diagram showing the configuration of a search server for image search according to the present invention.
3 is a block diagram illustrating a configuration of a user device for searching for an image according to the present invention.
4 is a table illustrating surrounding information extracted during an image search process in the method for image search according to an embodiment of the present invention.
5 is a flowchart illustrating an image retrieval process in the method for image retrieval according to the first embodiment of the present invention.
6 is a flowchart illustrating an image search process in the method for the image search according to the second embodiment of the present invention.
7 and 8 are exemplary diagrams for describing an image search word and a search process according to an exemplary embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS To make the features and advantages of the present invention more clear, the present invention will be described in more detail with reference to specific embodiments shown in the accompanying drawings.

However, in the following description and the accompanying drawings, detailed descriptions of well-known functions or configurations that may obscure the subject matter of the present invention will be omitted. In addition, it should be noted that like elements are denoted by the same reference numerals as much as possible throughout the drawings.

The terms or words used in the following description and drawings should not be construed as being limited to the common or dictionary meanings, and the inventors may appropriately define the concept of terms for describing their own invention in the best way. It should be interpreted as meanings and concepts corresponding to the technical idea of the present invention based on the principle that the present invention. Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical idea of the present invention, various modifications that can be substituted for them at the time of the present application It should be understood that there may be equivalents and variations.

In addition, terms including ordinal numbers, such as first and second, are used to describe various components, and are used only to distinguish one component from another component, and to limit the components. Not used. For example, without departing from the scope of the present invention, the second component may be referred to as the first component, and similarly, the first component may also be referred to as the second component.

In addition, when a component is referred to as being "connected" or "connected" to another component, it means that it may be connected or connected logically or physically. In other words, although a component may be directly connected or connected to other components, it should be understood that other components may exist in the middle and may be connected or connected indirectly.

In addition, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, the terms "comprises" or "having" described herein are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or the same. It is to be understood that the present invention does not exclude in advance the possibility of the presence or the addition of other features, numbers, steps, operations, components, parts, or a combination thereof.

In addition, the terms "?", "?", "Module", and the like described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. have.

In addition, "a" or "an", "one", "the" and the like shall also be indicated otherwise in the context of the present invention (particularly in the context of the following claims). Unless otherwise expressly contradicted by context, it may be used in the sense including both singular and plural.

In addition, embodiments within the scope of the present invention include computer readable media having or conveying computer executable instructions or data structures stored on computer readable media. Such computer readable media can be any available media that can be accessed by a general purpose or special purpose computer system. By way of example, such computer readable media may be in the form of RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or computer executable instructions, computer readable instructions or data structures. And may include, but are not limited to, any other medium that can be used to store or deliver certain program code means, and can be accessed by a general purpose or special purpose computer system. .

In the following description and claims, “network” is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules. When information is transmitted or provided to a computer system via a network or other (wired, wireless, or a combination of wired or wireless) communication connections, this connection can be understood as a computer-readable medium. Computer-readable instructions include, for example, instructions and data that cause a general purpose or special purpose computer system to perform a particular function or group of functions. The computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, or even source code.

In addition, the present invention relates to personal computers, laptop computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers In a network computing environment having various types of computer system configurations including pagers, etc., the present invention may be applied to provide an image retrieval service to the computer systems. The invention may also be practiced in distributed system environments where both local and remote computer systems that are linked via wired networks, wireless data links, or combinations of wired and wireless data links perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

First, an image retrieval system using surrounding information according to an embodiment of the present invention will be described.

1 is a diagram illustrating an image retrieval system using surrounding information according to an exemplary embodiment of the present invention. Referring to FIG. 1, the image search system includes a search server 100 and a user device 200 connected through a network NW.

The search server 100 and the user device 200 may transmit and receive data to and from each other through the network NW. For example, the communication network NW may include a wireless communication network such as a wireless local area network (WLAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA). Depending on how the system is implemented, it may also include wired communication networks such as Ethernet, xDSL (ADSL, VDSL), Hybrid Fiber Coaxial Cable (HFC), Fiber to The Curb (FTTC), and Fiber To The Home (FTTH).

In addition, the network (NW) used in the present invention may include, for example, a mobile communication network including a plurality of access networks (not shown) and a core network (not shown) connecting them. Here, the access network is a network that performs wireless communication by directly connecting to a terminal, for example, a plurality of base stations such as a base station (BS), a base transceiver station (BTS), a NodeB, an eNodeB, and a base station controller (BSC). It may be implemented as a base station controller such as a Radio Network Controller (RNC). In addition, as described above, the digital signal processing unit and the wireless signal processing unit, which are integrally implemented in the base station, are divided into digital units (hereinafter referred to as DUs and radio units (hereinafter referred to as RUs)). In addition, a plurality of RUs (not shown) may be installed in each of the areas, and a plurality of RUs may be connected to a centralized DU, and a core network (not shown) constituting a mobile network together with an access network may be connected to an external network. It serves to connect networks, for example other communication networks, such as the Internet.

As described above, the core network is a network system that performs main functions for mobile communication services such as mobility control and switching between access networks, and performs circuit switching or packet switching, and performs mobile network. Manage and control the flow of packets within In addition, the core network may manage inter-frequency mobility and may play a role for interworking with traffic in the access network and the core network and other networks such as the Internet network. The core network may further include a serving gateway (SGW), a PDN gateway (PGW), a mobile switching center (MSC), a home location register (HLR), a mobile mobility entity (MME), and a home subscriber server (HSS). It may be.

In addition, the network NW according to the present invention may include an internet network. The Internet network refers to a common public communication network, that is, a public network, through which information is exchanged according to the TCP / IP protocol. Through the network NW, the search server 100 and the user device 200 interoperate with each other to form a base image search system according to the present invention.

The user device 200 requests an image search by transmitting an image search word in the form of an image file to the search server 100 according to a user request. In this case, the image search word may include image data, and the image data may be a complex image including a plurality of searchable items. Here, an item means an object included in an image and having an independent meaning or name. The item may be, for example, a product sold in an online shopping mall, or may be nature or a person such as an animal or a plant. In such a composite image, a user may want to search for a specific item.

When the search server 100 receives the image search request, the search server 100 searches for the image search word. At this time, the search server 100 extracts surrounding information related to the search object from the background image and / or metadata of the image search word. The image search is performed in consideration of the surrounding information. For example, after defining a search range (eg, category, time, region) using surrounding information, and performing an image search, or after performing an image search, ranking of search results using the surrounding information I can adjust it. For reference, when the image search word is a composite image, the search server 100 selects a search target item desired by a user from among a plurality of items included in the image data. The selection of the item may be made through prediction based on a correlation between a user context and a category.

The search server 100 searches for the same or similar image as the image requested for the search based on the search target item, and provides the user device 200 with the same or similar image derived as a result of the image search.

Here, the search server 100 is an entity existing on the network, and serves as a web server, a database server, and an application server. According to a preferred embodiment, the search server 100 searches for and provides an image identical or similar to a web server providing a shopping mall web site, a database server storing an image of an item sold on the web site, and an image search word. It can act as an application server. In the following embodiment, it is assumed that the search server 100 is a server that provides an Internet shopping mall service that sells various products. Accordingly, the search server 100 may search for an image similar to an image search word among a plurality of images stored in a database. Such an image sold in an internet shopping mall may be an image of a product. Meanwhile, according to another exemplary embodiment, the search server 100 may search for images from a plurality of web pages by accessing another server, that is, a web server, through a network.

In addition, the user device 200 will be described as a representative example of a mobile communication terminal, but the terminal is not limited to the mobile communication terminal, but a variety of information communication devices, multimedia terminals, wired terminals, fixed terminals and IP (Internet Protocol) terminals, etc. Applicable to the terminal. For example, the terminal may be a mobile terminal having various mobile communication specifications such as a mobile phone, a portable multimedia player (PMP), a mobile internet device (MID), a smart phone, a tablet PC, a tablet PC, and an information communication device. Can be advantageously used. As another example, the terminal may be a notebook, a personal computer, or the like.

Then, the configuration of the search server 100 and the user device 200 according to an embodiment of the present invention will be described in detail. First, the configuration of the search server 100 according to an embodiment of the present invention will be described.

2 is a block diagram illustrating a configuration of a search server according to an embodiment of the present invention. Referring to FIG. 2, the search server 100 includes a communication module 110, a storage module 120, and a control module 130.

The communication module 110 is for communication with the user device 200. The communication module 110 connects to the network NW, and when the user device 200 connects to the search server 100 through the network NW, exchanging data including necessary information with the user device 200. Perform communication for When the communication module 110 receives data for transmission from the control module 130 to the user device 200, for example, a search result including a plurality of images, the communication module 110 configures and transmits the received data into a packet. In addition, the communication module 110 extracts the data, that is, the image search word from the packet received from the user device 200 and transmits it to the control module 130.

The storage module 120 stores a program and data necessary for the operation of the search server 100 and may be divided into a program area and a data area. The program area may store a program for controlling the overall operation of the search server 100, an operating system (OS) for booting the search server 100, an application for image search, and the like. The data area is an area in which data generated according to the operation of the search server 100 and data necessary for the operation of the search server 100 are stored. For example, the storage module 120 may store images for a plurality of items. Various data stored in the storage module 120 may be deleted, changed, or added according to a user's manipulation.

The control module 130 may control a general operation of the search server 100 and a signal flow between internal blocks of the search server 100 and may perform a data processing function for processing data. The control module 130 is preferably a microprocessor or a central processing unit (CPU). The control module 130 includes a peripheral information processing module 131, an image processing module 133, and an image retrieval module 135.

The peripheral information processing module 131 extracts peripheral information related to the input image search word. Specifically, the peripheral information processing module 131 separates the search target item and the background image which is the remaining area from the image search word, analyzes the background image, and analyzes the temporal or spatial or other information related to the search target item (for example, a person). Information) can be extracted. In addition, the peripheral information processing module 131 may analyze the metadata of the image search word and extract temporal, spatial and other information (personal information) related to the image search word.

In addition, the surrounding information processing module 131 extracts the surrounding information based on one of the background image and the metadata, and then corrects or supplements the extracted surrounding information based on the other of the background image and the metadata. Can be.

Accordingly, in the present invention, the surrounding information is information related to a search target item, and more specifically, one of temporal information, spatial information, and person information related to one or more of production, sale, and use of the search target item. It can contain more.

Referring to FIG. 4, the background image refers to an image other than a search target item among image data designated as an image search word. Through this background, temporal and spatial information may be extracted. For example, through the background, it is possible to extract rough spatial information such as whether the space where the image is captured is indoor, outdoor, outdoor, park, or street.In the case of outdoor, it is day or night without knowing the exact time. The schematic temporal information of can be extracted. In addition, when the item to be searched is a product such as clothing or a bag, it is possible to extract person information such as whether the person who wears the clothing or bag is a woman or a man or an approximate age group. Such information may be extracted through a learning algorithm through machine learning such as deep learning.

In addition, the metadata is data for recording information related to the data, and in the case of an image, may include information such as a photographing date, a photographing position, a photographing condition, a photographer, and the like. Accordingly, in the case of the present invention, as shown in FIG. 4, through the analysis of the metadata, accurate capturing time and location information can be extracted, and also the type information of the image which is an image search word, photographed user information, etc. Can be extracted.

The peripheral information extracted by the peripheral information processing module 131 is divided and stored for each image search word, as shown in FIG. 4.

The image processing module 133 specifies an item to be an image search target from the image search word. For example, the image processing module 133 may limit a product of a category sold in an online shopping mall to a search target item, and extract a product sold in an online shopping mall from among various objects included in the image search term as a search target item. . In addition, when the image search term is a composite image, the image processing module 133 extracts a plurality of items from the composite image, and one of them is selected from the user, or the user's purchase history or search is performed. One of a plurality of items may be selected as a search target item based on a user context such as user selection of a result.

The image search module 135 performs an image search based on the search target item selected by the image processing module 133. The operation of the control module 130 including the peripheral information processing module 131, the image processing module 133, and the image retrieval module 135 will be described in more detail below.

Next, the user device 200 according to an embodiment of the present invention will be described. 3 is a block diagram illustrating a configuration of a user device according to an exemplary embodiment of the present invention. Referring to FIG. 3, a user device 200 according to an embodiment of the present invention includes a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

The communication unit 210 is for communication with other devices including the search server 100 and performs communication through various communication connection methods. The communication unit 210 may communicate by selecting any one of various communication connection type communication functions. The communication unit 210 may be implemented by one module or a plurality of modules. For example, the communication unit 210 according to the WCDMA, LTE, LTE-A, etc. according to the broadband mobile communication method to access the network through the base station, according to the Wireless Local Area Network (WLAN) method using Wi-Fi (wireless fidelity) A wireless near field communication method for accessing a network through an access point (AP) and performing communication may be used. The communication unit 210 may include an RF transmitter for up-converting and amplifying a frequency of a transmitted signal, and an RF receiver for low noise amplifying and down-converting a received signal. Data received from the controller 250, for example, an image search word may be converted into a wireless signal and transmitted through a wireless channel. In addition, the communication unit 210 may receive a wireless signal including data, for example, an image search result, through the wireless channel and transmit the received data to the control unit 250.

The input unit 220 receives a key operation of the user for controlling the user device 200, generates an input signal, and transmits the generated input signal to the controller 250. The input unit 220 may include any one of a power key, a numeric key, and a direction key for power on / off, and may be formed as a predetermined function key on one surface of the user device 200. When the display unit 230 is formed of a touch screen, functions of various keys of the input unit 220 may be performed in the display unit 230. When the display unit 230 may perform all functions using only the touch screen, the input unit 220 may be omitted. It may be.

The display unit 230 visually provides a menu, input data, function setting information, and various other information of the user device 200 to the user. The display unit 230 outputs various screens such as a boot screen, a standby screen, and a menu screen of the user device 200. The display unit 230 may be formed of a liquid crystal display (LCD), organic light emitting diodes (OLEDs), active matrix organic light emitting diodes (AMOLEDs), and the like. Meanwhile, the display unit 230 may be implemented as a touch screen. In this case, the display unit 230 may include a touch sensor, and the controller 250 may detect a user's touch input through the touch sensor. The touch sensor may be configured as a touch sensor such as capacitive overlay, pressure, resistive overlay, infrared beam, or a pressure sensor. . In addition to the above sensors, all kinds of sensor devices capable of detecting contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor detects a user's touch input, generates a detection signal, and transmits the detected signal to the controller 250. The sensing signal may include coordinate data input by the user. When the user inputs the touch position movement operation, the touch sensor may generate a detection signal including coordinate data of the touch position movement path and transmit the detected signal to the controller 250.

The storage unit 240 stores a program and data necessary for the operation of the user device 200, and may be divided into a program area and a data area. The program area may store a program for controlling the overall operation of the user device 200, an operating system (OS) for booting the user device 200, an application program, and the like. The data area is an area in which data generated according to the use of the user device 200 is stored. In addition, the storage unit 240 may store various data generated according to the operation of the user device 200. The controller 250 may control an overall operation of the user device 200 and a signal flow between internal blocks of the user device 200, and may perform a data processing function of processing data. The controller 250 may be a central processing unit (CPU), an application processor, a graphic processor (GPU), or the like. When the image search word, for example, image data is input, the controller 250 transmits the input image data to the search server 100 through the communication unit 210, and when the search result is received from the search server 100, The search result may be displayed through the display unit 130.

Next, a method for image retrieval according to an embodiment of the present invention will be described.

5 is a flowchart illustrating an image search process according to a first embodiment of the present invention.

Referring to FIG. 5, the controller 250 of the user device 200 may request an image search by transmitting an image search word including image data to the search server 100 through the communication unit 210.

As such, when the control module 130 of the search server 100 receives an image search request from an arbitrary user device 200 through the communication module 110 (S110), the image data of the image search request for the image search is requested. In operation S120, the search target item is extracted, and peripheral information related to the search target item is extracted.

The search target item may be extracted as a searchable object among a plurality of objects included in the image. For example, an object corresponding to a product category sold in a designated online shopping mall among a plurality of objects included in an image may be extracted as a search target item. In this case, when the image corresponding to the image search word is a composite image including a plurality of items, the search server 100 may select one of the plurality of items as a search target item. The selection may be made according to a user input, or may be predicted based on a user context, and may be made based on the predicted category.

In the above description, the search item selection process through category prediction based on the user context will be described in more detail.

Assuming that the search server 100 is an online shopping mall server that sells products, the user device 200 accesses the search server 100 to purchase an item, stores an item of interest in a shopping cart (temporarily stores), You can search for the item you want to purchase. Whenever there is such an operation of the user device 200, the search server 100 identifies the time, the number of times of the purchase, temporary storage, and search, and the category to which the item belongs to the identifier or user identifier (eg, ID) of the user device 200. Can be mapped and stored. In this case, the plurality of user contexts may exemplify the number of purchases of the items by category, the number of times the items are included in the shopping cart, the number of times the items are searched by category, and the like. Such user context may be collected using cookies generated in a web browser.

As described above, the search server 100 extracts the user context, accumulates and stores the category of the searched image and the extracted user context as one set of training data in the storage module 120 and stores the user's interest. The relationship between the category of the item and the category of the item intended by the user at the search execution time Tin may be analyzed, and a search target item may be selected based on the correlation.

In addition, the search server 100 provides a search result including a plurality of searched images corresponding to the image search request to the user device 200, and the control unit 250 of the user device 200 receiving the search result is a display unit. A plurality of images that are search results are displayed at 230. In this case, the user may select an image desired by the user from among a plurality of images, and the image selected by the user is likely to be a user's preferred or desired image. Accordingly, when any one image is selected, the controller 250 detects the image through the input unit 220 or the display unit 230, and retrieves information about the image selected by the user through the communication unit 210. Can be delivered to. In addition, the image processing module 133 of the search server 100 classifies, accumulates and stores the images selected by the user among the plurality of images included in the search result, and groups the images searched by category according to the design classification. After creating a plurality of groups, a user's design preference can be determined according to the number of elements in each group. Based on these preferences, a user can select a high-preferred item among a plurality of items included in an image search term as a search target item. Can be.

When the search server 200 extracts a search target item from an image search word through the above-described various methods, the search server 200 extracts surrounding information related to the search target item (S120).

The extraction of the surrounding information may be performed by analyzing metadata included in the background image and / or image data which are images other than the search target item among the image search words.

In detail, the peripheral information processing module 131 of the search server 200 separates the input image search term into a search target item and the rest of the background image, analyzes the background image, and analyzes the background image in time or space or in relation to the search target item. Extract person information.

Here, the background image refers to a region other than a search target item among images designated as an image search word, and through this background, temporal and spatial information may be extracted.

For example, when an image as shown in FIG. 7 is inputted as an image search term and a searchable item includes a hat, a top, a bottom, and a bag, the search server 100 may search for the top, bottom, bottom, according to a user context. Among the bags, a target that is likely to be of interest to the user may be selected as a search target item. For example, when a bag among the hat, top, bottom, and bag is selected as a search target item, the remaining items and the background are separated into a background image. That is, when the bag is selected as the search target item, the search server 100 separates the person carrying the bag, the hat worn by the person, the top, the bottom, and the back background into the background image, and the background from the background image. Extract the information. Specifically, the gender and age group may be extracted from the image of the person carrying the bag, and the temporal information corresponding to the summer may be extracted according to the degree of exposure of the clothing, the top, and the bottom worn by the person. . Such information is used as the peripheral information of the bag which is the search target item.

In addition, the search server 100 may extract schematic spatial information such as whether the space in which the image is captured is indoors or outdoors, in the case of a park or a street in the outdoors, through analysis of the background image. In this case, even if the exact time is not known, rough temporal information about day or night may be extracted.

For example, an image of an image search word is a composite image including a plurality of searchable items indicated by H1, H2, B1, and B2 as shown in FIG. 8, and a bag indicated by B1 is a search target item based on a user's context. If selected, the search server 100 may separate all images other than the bag B1 into a background image. The image may be extracted from an outdoor distance from the background image, and information of a young woman may be extracted from the image of the person wearing the bag.

The surrounding information from the background image may be extracted through a learning algorithm through machine learning such as deep learning.

In addition, the peripheral information processing module 131 of the search server 100 may analyze metadata of the image search word to extract temporal, spatial and other information related to the image search word. The metadata is data for recording information related to data, and in the case of an image, may include information such as a photographing date, a photographing position, a photographing condition, a photographer, and the like. Accordingly, the peripheral information processing module 131 of the search server 100 may extract accurate photographing time and location information of the search target image through analysis of the metadata, and may also extract the type information of the image that has become an image search term. , Or photographed user information can be extracted.

In addition, the search server 100 may extract the surrounding information in consideration of the background image and the metadata. More specifically, the peripheral information processing module 131 of the search server 100 corrects or supplements the peripheral information with reference to the metadata of the image data, or analyzes the metadata of the image data to analyze the peripheral information. The information may be extracted and the surrounding information may be corrected or supplemented by analyzing a background image of the remaining image area except for the search target item among the image data.

As described above, when the search target item and its surrounding information are extracted through the processing of the image search word, the search server 100 analyzes the extracted surrounding information and specifies a search range desired by the user (S130). For example, in the image search word as shown in FIG. 7, when a bag is extracted as a search target item, the summer bag for women may be specified as a search range through analysis of the surrounding information. In addition, based on the person information, a search range may be specified as a summer bag for teenage women by specifying an age group.

In addition, in operation S130, the search server 100 compares the surrounding information extracted from the background image with the surrounding information extracted from the meta data and adjusts the two information if they are different. For example, based on the surrounding information extracted from the background image, the information extracted from the metadata may be used to supplement the surrounding information extracted from the background image. In detail, as a result of analyzing the background image, when temporal information cannot be extracted, relevant temporal information may be extracted and supplemented from metadata.

For example, as shown in FIG. 8, when the composite image is an image search word and the search target item is designated as the bag B1, the search server 100 may display a time of 'day' from the background image among images other than the bag B1. Information, and spatial information such as 'outdoor' and 'street' can be extracted, and the character information of 'young woman' can be extracted through analysis of the person wearing the bag B1. In this case, when more accurate time information is required than temporal information called 'day' or more accurate position information is needed than spatial information called 'distance', the search server 100 analyzes the metadata of the image, and corresponds to the metadata from the metadata. The temporal and spatial information can be supplemented from the photographing time and position information of the image. That is, the account, month, or year corresponding to the image may be extracted by referring to the time information of the metadata, and the location where the distance is located may be supplemented by referring to the spatial information of the metadata. In the case of FIG. 8, the spatial information “distance” may be supplemented with “New York street” or “London street” through analysis of metadata.

In addition, the search server 100 searches for an image similar to the search target item by performing an image search based on the extracted search target item (S140). The image search in step S140 may be performed through a conventional image search technique. For example, the similarity between the image search word and the images in the search DB may be calculated through a learning algorithm such as deep learning to extract an image having a similarity or more than a predetermined criterion.

Steps S130 and S140 may be performed at about the same time.

The search server 100 adjusts the search result of the search target item based on the search range specified through the analysis of the surrounding information (S150). In detail, the search server 100 may adjust the ranking of the plurality of images so that the image corresponding to the specified search range is exposed earlier among the plurality of images included in the search result.

Then, the control module 130 of the search server 100 transmits a search result including a plurality of images to the user device 100 through the communication module 110 (S260).

For example, in providing an image search result for a bag in the image search query as shown in FIG. 7, the search server 100 preferentially provides an image corresponding to the summer bag for women based on the surrounding information to the user. Can provide.

As another example, in providing an image search result for the bag B1 in the image search word as shown in FIG. 8, the search server 100 is related to 'a bag popular in New York' or a 'New York style bag'. A high bag image may be provided to the user first, or an image related to 'bags popular in New York at a certain age' may be provided to the user first.

When the control unit 250 of the user device 200 receives a search result through the communication unit 210, the control unit 250 displays a plurality of images included in the search result in a specified order through the display unit 230. Accordingly, the user may preferentially check an image corresponding to a specific search range based on the surrounding information, and thus may make the search result desired by the user more.

Next, an image search process according to the second embodiment of the present invention will be described with reference to FIG. 6.

As described above in the first embodiment, the control unit 250 of the user device 200 requests an image search by transmitting an image search word including image data to the search server 100 through the communication unit 210, and searching. When the control module 130 of the server 100 receives the image search request of the user device 200 through the communication module 110 (S210), the search server 100 of the image search request for the image search is requested. The image data is analyzed, the search target item is extracted, and peripheral information related to the search target item is extracted (S220).

Extraction of the search target item and the surrounding information may be performed in the same manner as in the first embodiment.

As described above, when the search target item and its surrounding information are extracted through the processing of the image search word, the search server 100 first analyzes the extracted surrounding information to specify a search range desired by the user (S230).

For example, in the image search word as shown in FIG. 7, when a bag is extracted as a search target item, the summer bag for women may be specified as a search range through analysis of the surrounding information. In addition, based on the person information, a search range may be specified as a summer bag for teenage women by specifying an age group. In operation S130, the search server 100 may compare the surrounding information extracted from the background image with the surrounding information extracted from the meta data and adjust the information when the two information are different. For example, based on the surrounding information extracted from the background image, the information extracted from the metadata may be used to supplement the surrounding information extracted from the background image. In detail, as a result of analyzing the background image, when temporal information cannot be extracted, relevant temporal information may be extracted and supplemented from metadata.

When the search range is specified through the analysis of the surrounding information as described above, the search server 100 searches for an image similar to the search target item by performing an image search based on the search target item within the specified search range ( S240).

The image search in step S240 may be performed through a conventional image search technique. For example, a similarity between the image search word and the images in the search DB may be calculated by using a learning algorithm such as deep learning, and the image may be extracted by having a similarity or more than a predetermined criterion. In this case, since the search range is limited based on the analysis result of the surrounding information, the search result can be derived more quickly through the limitation of the search range. Here, the search range may be set based on one or more of a temporal attribute, a spatial attribute, and a category attribute. Here, the temporal and spatial attributes may be time and space information related to the production of the search target item, or may be time and space information related to the use of the search target item. In addition, the category attribute may be subdivided into various characteristics such as age, gender, and design attribute of the person who is using the item.

The search server 100 stores and the image DB classifies the stored images into the above-described temporal attribute, spatial attribute, and category attribute, and extracts images corresponding to the specific search range among the classified images. .

The control module 130 of the search server 100 transmits a search result including a plurality of images to the user device 100 through the communication module 110 (S250).

When the control unit 250 of the user device 200 receives a search result through the communication unit 210, the control unit 250 displays a plurality of images included in the search result in a specified order through the display unit 230. Accordingly, the user may preferentially check an image corresponding to a specific search range based on the surrounding information, and thus may make the search result desired by the user more.

As described above, when the present invention provides an image search result in an information-rich modern society, it is possible to further refine the information by using the surrounding information extracted from the image requested to be searched without providing a user's intervention. have.

As described above, the specification includes the details of a number of specific implementations, but these should not be understood as being limited to the scope of any invention or claimable, but rather may be specific to a particular embodiment of a particular invention. It should be understood as a description of the features. Certain features that are described in this specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, while the features operate in a specific combination and may be initially depicted as so claimed, one or more features from the claimed combination may in some cases be excluded from the combination, wherein the claimed combination is a subcombination Or a combination of subcombinations.

Likewise, although the operations are depicted in the drawings in a specific order, it should not be understood that such operations must be performed in the specific order or sequential order shown in order to obtain desirable results or that all illustrated operations must be performed. In certain cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that it can.

Specific embodiments of the subject matter described in this specification have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order but still achieve desirable results. As an example, the process depicted in the accompanying drawings does not necessarily require that particular illustrated or sequential order to obtain the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

The foregoing description presents the best mode of the invention, and provides examples to illustrate the invention and to enable those skilled in the art to make and use the invention. The specification thus produced is not intended to limit the invention to the specific terms presented. Thus, while the present invention has been described in detail with reference to the examples described above, those skilled in the art can make modifications, changes and variations to the examples without departing from the scope of the invention.

Therefore, the scope of the present invention should be determined by the claims rather than by the described embodiments.

The present invention is applicable to the field of image retrieval, in performing image retrieval, extracts additional information related to items such as temporal information and spatial information from background images and metadata of images in addition to the items to be searched, By limiting the search range to perform a search, or by adjusting the ranking of the search results can further shape the user's confidence and satisfaction with the image search results provided to the user.

100: search server 110: communication module
120: storage module 130: control module
131: peripheral information processing module 133: image processing module
135: image search module 200: user device
210: communication unit 220: input unit
230: display unit 240: storage unit
250: control unit

Claims (12)

The device for image search,
Receiving an image search request comprising image data from a user device;
Extracting a search target item and surrounding information related to the search target item from the image data by the image search request;
Performing a search based on a search target item of the image data in consideration of the surrounding information; And
And providing a search result to the user device. The method of claim 1, wherein the extracting step
Extracting the remaining image area of the image data excluding the search target item as a background image, and extracting the surrounding information through analysis of the background image. . The method of claim 1, wherein the extracting step
Extracting metadata of the image data and extracting the peripheral information from the metadata. The method of claim 1, wherein the extracting step
After extracting the remaining image area of the image data excluding the search target item as a background image, extracting the surrounding information through analysis of the background image, and correcting or supplementing the surrounding information with reference to metadata of the image data. Method for image search using the surrounding information, characterized in that. The method of claim 1, wherein the extracting step
Extracting the peripheral information by analyzing metadata of the image data,
And analyzing or supplementing the surrounding information by analyzing a background image of the remaining image area of the image data except for the search target item. The method of claim 1, wherein the surrounding information is
And at least one of temporal information, spatial information, and person information related to at least one of production, sale, and use of the searched item. The method of claim 1, wherein performing the search comprises:
Analyzing the surrounding information to specify a search range;
Performing an image search based on the search target item; And
And modifying a search result of the image search based on the specific search range. The method of claim 1, wherein performing the search comprises:
Analyzing the surrounding information to specify a search range; And
And performing an image search on the basis of the search target item within the search range. An image processing module configured to analyze image data requested for an image search and to select a search target item among a plurality of searchable objects included in the image data;
A peripheral information processing module for extracting peripheral information related to the search target item by analyzing at least one of a background image excluding the search target item among the image data and metadata of the image data; And
And an image search module configured to search for an image similar to the image data based on the search target item in consideration of the surrounding information. The method of claim 9, wherein the peripheral information processing module,
After extracting the surrounding information by analyzing a background image among the image data, correcting or supplementing the surrounding information with reference to the metadata of the image data,
And extracting the surrounding information by analyzing the metadata of the image data, and correcting or supplementing the surrounding information by analyzing the background image. The method of claim 9, wherein the surrounding information is
And at least one of temporal information, spatial information, and person information related to at least one of production, sale, and use of the searched item. The method of claim 9, wherein the image search module
Analyze the surrounding information to specify a search range, perform an image search based on the search target item, and then modify a search result of the image search based on the specific search range;
The apparatus for searching for an image using the surrounding information, characterized by specifying the search range by analyzing the surrounding information, and performing an image search based on the search target item within the specific search range.

Images