KR102651950B1 - A method for sketch based image searching and an apparatus therefor - Google Patents

Abstract

The present invention relates to a method of providing a sketch for sketch-based image search and a device therefor. The present invention includes the steps of receiving a request for a sketch for a sketch-based image search by a sketch providing module of a user device, a step of extracting user context from a predetermined period of time prior to the request by the sketch providing module, and retrieving images that have already been performed. Deriving a category by inputting the collected user context into a correlation equation derived according to the correlation between the resulting category and the user context, and prioritizing the derived category and a plurality of sketches belonging to the derived category. A method for providing a sketch for sketch-based image search, which includes the step of listing and displaying the image, and a device therefor are provided.

Description

{A method for sketch based image searching and an apparatus therefor}

The present invention relates to image search, and more specifically, by providing a sketch to help the user sketch, and by providing a plurality of sketches according to the user context (User Context) representing the behavior pattern according to the user's use of the user device. It relates to a method and device for sketch-based image search that facilitates sketch-based image search.

Image search can be broadly divided into two types depending on the method and purpose of finding images. The first is to find related images using text search terms. For example, if you enter the keyword 'flower', you can see photos of various types of flowers. This type is a traditional search method that has been around since the early days of the Internet. The second is that when you enter an image search term, which is an image file, the image file is analyzed to find the same or similar images. This method emerged as computer analysis technology and Internet technology developed. Compared to the keyword method, it has the advantage of allowing users to intuitively search for images.

Korean Patent Publication No. 2013-0059058, published on June 5, 2013 (Name: Apparatus and method for pre-processing query images and search target images in a content-based image searcher using sketch query)

The purpose of the present invention is to provide a sketch when a user performs a search based on a sketch input, and to predict the sketch of the category the user needs by analyzing the user context representing the behavior pattern according to the user's use of the user device. The goal is to provide a method and device for sketch-based image search.

A method for sketch-based image search according to a preferred embodiment of the present invention for achieving the above-described object includes the steps of: receiving a request for sketch-based image search by a sketch-based image search device; displaying a plurality of sketches according to the request; When one of the plurality of drafts is selected, displaying the draft in a sketch area of the screen; When detecting a user input on the sketch area, displaying a sketch according to the trace of the user input along with the sketch according to the detected user input; and transmitting the sketch and the sketch to a search server to request an image search.

In the method for sketch-based image search according to the present invention, displaying the plurality of sketches includes: extracting, by the device, user context prior to a predetermined period of time from the time of the request; Deriving a category by inputting the collected user context into a correlation equation derived according to the correlation between the category of the image search result performed by the device and the user context; and displaying, by the sketch providing module, the derived category and a plurality of drafts belonging to the derived category in order of priority.

In addition, the sketch-based image search method according to the present invention includes the steps of collecting user context, categories of image search results, and valid images whenever image search is performed by the device before the request receiving step; generating, by the device, a correlation equation defining a correlation between a category of the image search result and a user context; and generating, by the device, a sketch from the collected valid images.

In the sketch-based image search method according to the present invention, the step of collecting the user context, the category of the image search result, and the valid image includes the device retrieving the image from a predetermined period of time based on the image search time. collecting a plurality of user contexts up to a point in time; detecting and storing, by the device, at least one valid image for a predetermined period of time after the image retrieval point; and determining, by the device, a category of the detected valid image.

In addition, the step of generating the correlation equation includes, after determining the category, the device performing an image search among a plurality of user contexts collected from a predetermined period of time to the time of the image search. Deriving a user context in which the correlation coefficient with the occurring event is greater than or equal to a predetermined value; selecting, by the device, a user context among the plurality of collected user contexts where the correlation coefficient between two or more user contexts is greater than or equal to a predetermined value and the correlation coefficient with the event in which the image search occurs is the highest; and generating, by the device, a correlation equation through linear regression analysis for the selected user context and the category.

In addition, in the step of generating a correlation equation through the linear regression analysis, C is the probability that a category is selected, i is the index of the category, K is the number of selected user contexts, X1, , Xk are different user contexts, and a0, a1, a2, a3, ..., an are weights, and the device Each time an image search is performed, the values of the collected categories and contexts are input as training data to construct a plurality of simultaneous equations, the weights are calculated with the solution of the constructed simultaneous equations, and the calculated weights are applied to the equation. Thus, a correlation equation can be created.

In the step of generating a draft from the collected valid images, the draft providing module groups the plurality of valid images collected for each category into a plurality of types according to the characteristics of the shape to generate a plurality of valid image groups. ; generating, by the sketch providing module, a plurality of binary images consisting only of dots and lines by detecting outlines of each of the plurality of valid images belonging to the valid image group according to a morphology analysis technique; A step of the sketch providing module overlapping the plurality of binary images to generate a draft according to the degree of overlap of points and lines of the plurality of overlapping binary images; and determining, by the draft providing module, the priority of the generated draft according to the number of valid images belonging to the valid image group.

In addition, the present invention provides a method for sketch-based image search by a search server as another solution to the above-described problem. The method includes the steps of the search server receiving a sketch and the draft along with an image search request from a user device; The search server deriving differences between the sketch and the draft as a feature image; Searching the search server for a plurality of images identical or similar to the sketch; generating a search result in which the search server sorts the plurality of searched images in order of high similarity to the feature image; and transmitting, by the search server, the search results to the user device.

In addition, the present invention includes a communication unit that connects to a network and performs communication; a display unit for displaying a screen including a sketch area and a draft area; When a sketch-based image search is requested, the user context from a predetermined period of time prior to the request is extracted, and the extracted user context is entered into a correlation equation derived according to the correlation between the user context and the category of the previously performed image search results. a sketch providing module that derives a category through input, arranges the derived category and a plurality of drafts belonging to the derived category in order of priority, and displays them in the draft area; And when one of the plurality of sketches is selected, the sketch is displayed in the sketch area of the screen, and when a touch input on the sketch area is detected, a trace of the touch input is displayed along with the sketch according to the detected touch input. , a sketch search module that transmits the sketch and the sketch to a search server through the communication unit; providing a device for sketch-based image search, comprising:

Here, whenever an image search is performed, the sketch providing module collects a plurality of user contexts from a predetermined period before the image search time to the image search time, and at least one user context during a predetermined period after the image search time. Detecting a valid image, determining a category of the detected valid image, deriving a user context whose correlation coefficient with the category is more than a predetermined value among the plurality of collected user contexts, and 2 of the plurality of collected user contexts The user context in which the correlation coefficient between the above user contexts is greater than a predetermined value and the highest correlation coefficient with the category may be selected, and a correlation equation may be generated through linear regression analysis of the selected user context and the category.

In addition, in the sketch providing module, C is the probability of a category being selected, i is the index of the category, K is the number of selected user contexts, and X1, X2, , a0, a1, a2, a3, ..., when an is a weight, the equation Each time an image search is performed, the values of the collected categories and contexts are input as training data to construct a plurality of simultaneous equations, the weights are calculated using the solutions of the constructed simultaneous equations, and the calculated weights are applied to the equation. A correlation equation can be created.

In addition, the sketch providing module generates a valid image group by grouping a plurality of valid images collected for each category, and creates an outline according to a morphology analysis technique for each of the plurality of valid images belonging to the valid image group. Detection generates a plurality of binary images consisting of only dots and lines, overlapping the plurality of binary images to generate a sketch according to the intensity of the overlapped dots and lines, and calculating the number of valid images belonging to the valid image group. Accordingly, the priority of the generated sketch can be determined.

In addition, the present invention includes a communication module that connects to a network and performs communication; And when receiving a sketch and a draft from the user device, derive the difference between the sketch and the draft as a feature image, search for a plurality of images identical or similar to the sketch, and compare the retrieved plurality of images with the feature image. An image search module that generates search results sorted in order of high similarity and transmits the search results to the user device through the communication module. Provides a search server for sketch-based image search, comprising: do

According to the present invention, in order to provide a sketch-based image search that performs a sketch by referring to the sketch to the user, the sketch of the category needed by the user is predicted by analyzing the user context representing the user's behavior pattern, and the sketch-based image search is performed. By providing a predicted sketch during a search, the sketch the user wants can be provided without the user having to spend time searching for the sketch.

1 is a diagram for explaining a system that provides sketch-based image search according to an embodiment of the present invention.
Figure 2 is a block diagram for explaining the configuration of a search server according to an embodiment of the present invention.
Figure 3 is a block diagram for explaining the configuration of a user device according to an embodiment of the present invention.
Figure 4 is a flowchart explaining a method of providing a sketch for sketch-based image search according to an embodiment of the present invention.
Figure 5 is a flowchart illustrating a method of performing sketch-based image search by providing a sketch according to an embodiment of the present invention.
Figure 6 is a flowchart for explaining a data collection method according to an embodiment of the present invention.
Figure 7 is a diagram for explaining a data collection method according to an embodiment of the present invention.
Figure 8 is a flowchart illustrating a method for generating a correlation equation according to an embodiment of the present invention.
Figure 9 is a flowchart illustrating a method of generating a sketch for each category according to an embodiment of the present invention.
10 and 11 are diagrams for explaining a method of generating a sketch for each category according to an embodiment of the present invention.
Figure 12 is a flowchart for explaining a method of displaying a sketch according to an embodiment of the present invention.
Figure 13 is an example screen for explaining a method of displaying a draft according to an embodiment of the present invention.
Figure 14 is a flowchart illustrating a method of creating a sketch and transmitting the sketch as an image search word according to an embodiment of the present invention.
Figure 15 is an example screen for explaining a method of creating a sketch according to an embodiment of the present invention.
Figure 16 is a flowchart for explaining a sketch-based image search method according to an embodiment of the present invention.
Figure 17 is a diagram for explaining a sketch-based image search method according to an embodiment of the present invention.

In order to make the characteristics and advantages of the problem-solving means of the present invention clearer, the present invention will be described in more detail with reference to specific embodiments of the present invention shown in the attached drawings.

However, detailed descriptions of known functions or configurations that may obscure the gist of the present invention are omitted in the following description and attached drawings. Additionally, it should be noted that the same components throughout the drawings are indicated by the same reference numerals whenever possible.

Terms or words used in the following description and drawings should not be construed as limited to their usual or dictionary meanings, and the inventor may appropriately define the concept of the term to explain his or her invention in the best way. It must be interpreted with meaning and concept consistent with the technical idea of the present invention based on the principle that it is. Therefore, the embodiments described in this specification and the configuration shown in the drawings are only one of the most preferred embodiments of the present invention, and do not represent the entire technical idea of the present invention, so at the time of filing the present application, various alternatives may be used to replace them. It should be understood that equivalents and variations may exist.

In addition, terms containing ordinal numbers, such as first, second, etc., are used to describe various components, and are used only for the purpose of distinguishing one component from other components and to limit the components. Not used. For example, the second component may be referred to as the first component without departing from the scope of the present invention, and similarly, the first component may also be referred to as the second component.

In addition, when a component is referred to as being “connected” or “connected” to another component, it means that it is or can be connected logically or physically. In other words, a component may be directly connected or connected to another component, but it should be understood that other components may exist in the middle and may be indirectly connected or connected.

Additionally, the terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In addition, terms such as "comprise" or "have" used in the specification are intended to designate the presence of features, numbers, steps, operations, components, parts, or a combination thereof described in the specification, but are not intended to indicate the presence of one or more of the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. It should be understood that this does not exclude in advance the presence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof.

In addition, terms such as "unit", "unit", and "module" used in the specification refer to a unit that processes at least one function or operation, which is implemented as hardware, software, or a combination of hardware and software. It can be.

Additionally, the words “a or an,” “one,” “the,” and similar words may indicate otherwise herein in the context of describing the invention (particularly in the context of the claims below). It may be used to include both singular and plural terms, unless clearly contradicted by the context.

Additionally, embodiments within the scope of the present invention include computer-readable media having or transmitting computer-executable instructions or data structures stored on the computer-readable media. Such computer-readable media may be any available media that can be accessed by a general-purpose or special-purpose computer system. By way of example, such computer-readable media may include RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or in the form of computer-executable instructions, computer-readable instructions or data structures. It may be used to store or transmit certain program code means, and may include, but is not limited to, a physical storage medium such as any other medium that can be accessed by a general purpose or special purpose computer system. .

In the following description and claims, “network” is defined as one or more data links that enable the transfer of electronic data between computer systems and/or modules. When information is transmitted or provided to a computer system over a network or other communication connection (wired, wireless, or a combination of wired and wireless), this connection may be understood as a computer-readable medium. Computer-readable instructions include, for example, instructions and data that cause a general-purpose or special-purpose computer system to perform a particular function or group of functions. Computer-executable instructions may be binary, intermediate format instructions, for example, assembly language, or even source code.

In addition, the present invention is applicable to personal computers, laptop computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, and pagers. In a network computing environment with various types of computer system configurations including pagers, etc., it can be applied to providing advertisements to the computer systems. The invention may also be practiced in distributed system environments where tasks are performed by both local and remote computer systems that are linked over a network via wired data links, wireless data links, or a combination of wired and wireless data links. In a distributed system environment, program modules may be located in local and remote memory storage devices.

First, a system that provides sketch-based image search according to an embodiment of the present invention will be described. 1 is a diagram for explaining a system that provides sketch-based image search according to an embodiment of the present invention. Referring to FIG. 1, the sketch-based image search system includes a search server 100 and a user device 200.

The search server 100 and the user device 200 can transmit and receive data to each other through a network (NW). For example, the network (NW) may include wireless communication networks such as Wireless LAN (WLAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA), Depending on how the system is implemented, it may include wired communication networks such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coaxial Cable), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home).

In addition, the network (NW) may include, for example, a mobile communication network consisting of a plurality of access networks (not shown) and a core network (not shown) connecting them. Here, the access network is a network that performs wireless communication by directly connecting to the terminal, for example, a number of base stations such as BS (Base Station), BTS (Base Transceiver Station), NodeB, eNodeB, etc., and BSC (Base Station Controller) , can be implemented with a base station controller such as RNC (Radio Network Controller). In addition, as described above, the digital signal processing unit and the wireless signal processing unit, which were integrated into the base station, were divided into a digital unit (hereinafter referred to as DU) and a radio unit (hereinafter referred to as RU), respectively, and were divided into multiple units. It can also be configured by installing a number of RUs (not shown) in each area and connecting the multiple RUs to a centralized DU. In addition, the core network (not shown) that constitutes the mobile network together with the access network is connected to the access network and the external network. It plays a role in connecting other communication networks, such as the Internet network.

As explained above, this core network is a network system that performs major functions for mobile communication services such as mobility control and switching between access networks, and performs circuit switching or packet switching, and is a mobile network system that performs circuit switching or packet switching. Manage and control packet flow within. Additionally, the core network may manage inter-frequency mobility and play a role in interoperating with traffic within the access network and core network and with other networks, such as the Internet network. This core network will further include Serving GateWay (SGW), PDN GateWay (PGW), Mobile Switching Center (MSC), Home Location Register (HLR), Mobile Mobility Entity (MME), and Home Subscriber Server (HSS). It may be possible.

Additionally, the network (NW) according to the present invention may include an Internet network. The Internet network refers to a normal open communication network, that is, a public network, in which information is exchanged according to the TCP/IP protocol. Through this network (NW), the search server 100 and the user device 200 interoperate to form a sketch-based image search system according to the present invention.

The user device 200 provides a sketch based on the user context, creates a sketch according to the user's touch input, and provides the generated sketch to the search server 100 as an image search term. Then, the search server 100 performs an image search for the sketch image, searches for an image identical or similar to the sketch, and provides the same or similar image derived as a result of the image search to the user device 200.

Here, the search server 100 is an entity existing on the network and performs the role of an application server. This search server 100 can connect to another server, that is, a web server, through a network and search for images from a plurality of web pages.

In addition, the user device 200 is described as a representative example of a mobile communication terminal, but the terminal is not limited to mobile communication terminals, and includes various information and communication devices, such as multimedia terminals, wired terminals, fixed terminals, and IP (Internet Protocol) terminals. It can be applied to terminals. For example, the terminal is a mobile terminal with various mobile communication specifications such as a mobile phone, PMP (Portable Multimedia Player), MID (Mobile Internet Device), smart phone, tablet PC, phablet PC, and information and communication devices. It can be used to advantage. As another example, the terminal may be a laptop, personal computer, etc.

Next, we will look at the configuration of the search server 100 and the user device 200 according to an embodiment of the present invention in more detail. First, the configuration of the search server 100 according to an embodiment of the present invention will be described. Figure 2 is a block diagram for explaining the configuration of a search server according to an embodiment of the present invention. Referring to FIG. 2, the search server 100 includes a communication module 110, a storage module 120, and a control module 130.

The communication module 110 is for communication with the user device 200. The communication module 110 performs communication to exchange data including necessary information with the user device 200 when the user device 200 connects to the search server 100 through the network (NW). Specifically, when the communication module 110 receives data to be transmitted from the control module 130 to the user device 200, it configures the received data into packets and transmits them. Additionally, the communication module 110 extracts data from packets received from the user device 200 and transmits the extracted data to the control module 130.

The storage module 120 serves to store programs and data necessary for the operation of the search server 100, and can be divided into a program area and a data area. The program area can store programs that control the overall operation of the search server 100, an operating system (OS) that boots the search server 100, and applications for image search. The data area is an area where data generated according to the operation of the search server 100 and data necessary for the operation of the search server 100 are stored. Various types of data stored in the storage module 120 can be deleted, changed, or added according to user operations.

The control module 130 controls the overall operation of the search server 100 and signal flow between internal modules of the search server 100, and can perform a data processing function to process data. This control module 130 is preferably a central processing unit (CPU) or a processor.

The control module 130 includes an image search module 300. According to one embodiment, when the image search module 300 receives a sketch as an image search term, it searches for the same or similar image as the sketch, and sends the image according to the search result to the user device 200 through the communication module 110. send. In particular, when the image search module 300 receives a draft or a sketch as an image search term, it analyzes the difference between the sketch and the sketch to derive a feature image, searches for images that are the same or similar to the sketch, and then searches for images that are identical or similar to the sketch, and then retrieves multiple images according to the search results. The images are sorted in descending order of similarity to the feature image and transmitted to the user device 200 through the communication module 110. The operation of the control module 130 including the image search module 300 will be described in more detail below.

Next, the user device 200 according to an embodiment of the present invention will be described. Figure 3 is a block diagram for explaining the configuration of the user device 200 according to an embodiment of the present invention. Referring to Figure 3, the user device 200 according to an embodiment of the present invention includes a communication unit 210, a location information receiver 220, a sensor unit 230, a camera unit 240, an input unit 250, and a display unit ( 260), an audio unit 270, a storage unit 280, and a control unit 290.

The communication unit 210 is for communication with other devices, including the search server 100, and performs communication through various communication connection methods. This communication unit 210 can communicate by selecting any one of the communication functions of various communication connection methods. This communication unit 210 may be implemented as one module or multiple modules. For example, the communication unit 210 uses a broadband mobile communication method that connects to a network through a base station according to standards such as WCDMA, LTE, and LTE-A, and a wireless local area network (WLAN) method that uses Wi-Fi (wireless fidelity). A wireless short-range communication method that performs communication by connecting to a network through an access point (AP), a P2P method that directly connects and communicates with other devices in the ISM (industrial scientific and medical equipment) band such as Bluetooth and infrared communication, etc. can be used. The communication unit 210 may be composed of an RF transmitter that up-converts and amplifies the frequency of a transmitted signal, and an RF receiver that amplifies the received signal with low noise and down-converts the frequency. Additionally, the communication unit 210 may receive a wireless signal containing data through a wireless channel and transmit it to the control unit 290. Additionally, data received from the control unit 290 can be converted into a wireless signal and transmitted through a wireless channel.

The location information receiver 220 is for receiving location information of the user device 100. For example, the location information receiver 220 continuously receives GPS signals from GPS satellites (not shown) and extracts location information from the received GPS signals. Then, the location information receiver 220 can transmit the location information to the control unit 290. This location information may be coordinates such as latitude, longitude, and altitude.

The sensor unit 230 is used to detect displacement and speed of movement of the user device 200. The sensor unit 230 may detect the displacement of the movement of the user device 200 and provide the detected displacement of the movement to the control unit 290 through coordinate data. Coordinates provided by the sensor unit 230 may provide displacement from an arbitrary reference position in a three-dimensional rectangular coordinate system (x, y, z) or latitude, longitude, and altitude. Additionally, the coordinates provided by the sensor unit 230 may provide displacements of yaw, pitch, and roll. Moreover, the sensor unit 230 can detect the speed of this movement and provide the speed to the control unit 290. The sensor unit 230 may be implemented using one or more sensors, and these sensors may be an accelerometer, a gyroscope, or a magnetometer.

The camera unit 240 is used to capture images. This camera unit 240 may be composed of a lens, an actuator, a filter, an image sensor, and an image processor (ISP, Image Signal Processor). The image sensor can be composed of a CCD (Charged Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor), and receives light reflected from the subject through a lens and converts it into an electrical signal. The image processor (ISP, Image Signal Processor) converts the electrical signal output from the image sensor into a digital sequence and outputs it as image data. In addition, the camera unit 240 controls the actuator through an auto focus (AF) and/or optical image stabilization (OIS) driver to perform auto focus (AF) and/or optical image stabilization (OIS). It can be done.

The input unit 250 receives the user's key operations to control the user device 200, generates an input signal, and transmits it to the control unit 290. The input unit 250 may include one of a power key, a number key, and a direction key for powering on/off, and may be formed as a predetermined function key on one side of the user device 200. If the display unit 260 is made of a touch screen, the functions of various keys of the input unit 250 can be performed on the display unit 260, and if all functions can be performed only by the touch screen, the input unit 250 can be omitted. It may be possible.

The display unit 260 visually provides menus, input data, function setting information, and various other information of the user device 200 to the user. The display unit 260 performs the function of outputting various screens, such as a boot screen, standby screen, and menu screen, of the user device 200. The display unit 260 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like.

Meanwhile, the display unit 260 is a component for outputting visual information of the user device 200 and can be implemented with various display devices. More specifically, in one embodiment of the present invention, the display unit 260 is implemented as a touch screen. In this case, the display unit 260 includes a touch sensor, and the control unit 290 can detect the user's touch input through the touch sensor. The touch sensor may be composed of a touch sensor such as capacitive overlay, pressure type, resistive overlay, or infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all types of sensor devices that can detect contact or pressure of an object can be used as the touch sensor of the present invention. The touch sensor detects the user's touch input, generates a detection signal, and transmits it to the control unit 290. These detection signals may include coordinate data where the user inputs a touch. When a user inputs a touch position movement operation, the touch sensor may generate a detection signal including coordinate data of the touch position movement path and transmit it to the control unit 290.

The display unit 260 according to an embodiment of the present invention may display an image search word input screen for receiving a search word image input through a sketch through a touch screen. Additionally, the display unit 260 may display a sketch to guide sketch input on the image search word input screen. When a touch input is made, the display unit 260 detects the user's touch input on the image search word input screen and provides the coordinates of the touch input, coordinate data of the touch location movement path, etc. to the control unit 290.

The audio unit 270 is for receiving and outputting audio signals. This audio unit 270 includes a microphone (MIK) and a speaker (SPK). That is, the audio unit 270 can receive audio signals such as music and voice through a microphone (MIK), and output audio signals such as music and voice through a speaker (SPK).

The storage unit 280 serves to store programs and data necessary for the operation of the user device 200, and can be divided into a program area and a data area. The program area can store programs that control the overall operation of the user device 200, an operating system (OS) that boots the user device 200, and application programs. The data area is an area where data generated according to the use of the user device 200 is stored. Additionally, the storage unit 280 may store various types of data generated according to the operation of the user device 200, such as context, action logs, sketches, etc.

The control unit 290 controls the overall operation of the user device 200 and signal flow between internal blocks of the user device 200, and may perform a data processing function to process data. This control unit 290 may be a central processing unit (CPU), an application processor, a graphics processor (GPU), or the like.

The control unit 290 includes a clock module 291. This clock module 291 includes a crystal oscillator and calculates the time at the current location based on standard time.

When a user wants to perform a sketch-based image search, the control unit 290 analyzes the user context and provides a suitable sketch to the user through the display unit 260 so that the user can easily input a sketch. For this purpose, the control unit 290 further includes a sketch providing module 400 and a sketch search module 500.

The sketch providing module 400 analyzes user context from past image search cases and derives a correlation equation that defines the correlation between the category of the image search result and the user context. And when a user requests a sketch, the sketch providing module 400 extracts the user context, inputs the extracted user context into a correlation equation, and predicts the current user's desired category as a solution to the correlation equation.

When one of a plurality of drafts is selected by the user, the sketch search module 500 displays the draft in the sketch area of the screen and detects a touch input on the sketch area where the draft is displayed to create a sketch. It is created, and the generated sketch and displayed sketch are transmitted to the search server 100 through the communication unit 210.

In addition, although not shown, the user device 200 according to an embodiment of the present invention includes a storage medium insertion unit that allows data storage by inserting an external storage medium such as a memory card, and a connection for data exchange with an external digital device. It may be provided with a terminal, a terminal for power supply or charging, etc. According to the convergence trend of digital devices, there are so many variations of portable devices that it is impossible to list them all, but units equivalent to the above-mentioned units are additionally included in the user device 200 according to the present invention. Anyone with ordinary knowledge in the technical field will be able to easily understand that it can be configured.

Next, a method for providing a sketch for sketch-based image search according to an embodiment of the present invention will be described. Figure 4 is a flowchart explaining a method of providing a sketch for sketch-based image search according to an embodiment of the present invention.

Before explaining FIG. 4, the terms 'user context', 'effective image', and 'category' according to an embodiment of the present invention will be explained.

The context is the user device 200, that is, the communication unit 210, the location information receiver 220, the sensor unit 230, the camera unit 240, the input unit 250, the display unit 260, the audio unit 270, and This refers to information generated according to the use of the storage unit 280 and the control unit 290. This context includes a communication unit 210, a location information receiver 220, a sensor unit 230, a camera unit 240, an input unit 250, a display unit 260, an audio unit 270, a storage unit 280, and a control unit. (290) Includes a preliminary context that can be obtained from itself and a composite context (Integrated context) that can be obtained by combining two or more basic contexts.

For example, the default context is: Call records, data transmission and reception records, etc. that can be obtained through the communication unit 210 may serve as context. Location information obtainable through the location information receiver 220, sensor information obtainable through the sensor unit 230, input signal obtainable through the input unit 250 or display unit 260, touch input signal, and audio unit. Output audio signal that can be obtained through the speaker (SPK) of 270, input audio signal that can be obtained through the microphone (MIC), stored data obtained through the storage unit 280, and execution of the application executed in the control unit 290. Records, etc. can be the default context.

For example, the composition context is as follows: The time obtained through a clock including a crystal oscillator of the control unit 290, the location information (coordinates) obtained through the location information receiver 220, and the electronic map stored in the storage unit 280 are combined to collect, Instead of location information that is simply coordinates, such as workplace, school, library, cafe, restaurant, gym, subway station, or airport, location information indicating the location where the user is located can be obtained as a synthetic context.

As another example, time obtained through a clock including a crystal oscillator of the control unit 290, location information (coordinates) obtained through the location information receiver 220, electronic map and sensor unit stored in the storage unit 280 Sensor information such as speed, acceleration, and altitude obtained through (230), input/output audio signals obtained through the audio unit 270, call or communication records obtained through the communication unit 210, and control unit 290. By combining the app execution records obtained through the application, behavioral information representing the user's current actions such as waking up, going to bed, going to work, leaving work, studying, exercising, eating, shopping, traveling, studying, playing games, listening to music, etc., is synthesized into a context. You can get it.

User context according to an embodiment of the present invention refers to a composite context that can be obtained by combining two or more basic contexts. This user context is used to analyze user behavior patterns by combining multiple basic contexts. The values of these user contexts are expressed as probabilities.

For example, in the case of typical office workers, they wake up in the morning and move from their residence to their workplace before commuting time. When they arrive at their workplace, they are automatically connected to the secure WiFi at their workplace. And office workers do not leave the work area much before leaving work, but when it is time to leave work, they move from their work place to their residence. When they arrive at their residence, they are automatically connected to their residence's secure WiFi. And since the user sleeps during the night, there is no change in the position and speed of the user device 200.

The user's residence and workplace can be estimated from the location information of the location information receiver 220, the time information of the control unit 290, the speed information of the sensor unit 230, and the WiFi connection information of the communication unit 210. . Additionally, commuting time can be estimated by specifying the time zone between residence and work location. Also, when moving between residence and work place, the means of transportation for commuting to and from work can be estimated using speed information and location information such as electronic subway map, electronic bus route map, etc. For example, when the user context is the commuting time, the commuting time is different for each user and is therefore a value estimated from a plurality of basic contexts over a predetermined period of time. Therefore, these user contexts are expressed as probabilities. For example, if the probability that it is the current commute time is 91%, the corresponding user context can be expressed as 0.910.

Image search is performed by searching an image using a search term, and multiple images that are the same or similar to the image being the search term are provided. At this time, the valid image refers to the image selected by the user among a plurality of images provided as an image search result.

The category (search category) refers to the field to which the valid image belongs. Image search results can be provided separately by category, and the valid image also belongs to one category. For example, when 'keyboard' is entered as a text search term, the user's intention or interest may be the keyboard, which is a peripheral device of a personal computer, or the keyboard, which is a musical instrument. In other words, the keyboard may belong to the category 'computer peripherals' or the category 'musical instruments'. Categories may be continuously added or changed over time. For example, when a search is performed using 'one piece' as a search term, 'one piece' may usually belong to the category 'women's clothing'. However, due to the popularity of the cartoon 'One Piece', when an image search is performed using the search term 'One Piece', images related to the cartoon 'One Piece' may be placed at the top of the search result priority. In this way, even for the same search term, completely different things are searched depending on the category. Therefore, when the user selects a valid image, the category of the valid image can be derived.

Therefore, whenever an image search is performed to predict which category the user wants to search for, the user device 200 uses data to generate a correlation equation in step S110, including user context and image, including basic and synthetic user context. A plurality of valid images according to search results can be collected, and categories can be derived from the collected plurality of valid images. It should be noted that this step S110 is continuously performed repeatedly each time an image search is performed.

When a sufficiently large number of data is accumulated in step S110, the user device 200 generates a correlation equation defining the correlation between the user context and the category in step S120. This correlation equation is created through linear regression analysis.

Next, in step S130, the user device 200 generates a plurality of sketches from a plurality of valid images for each category collected in step S110, maps them to each category, and stores them.

As described above, we will explain how to perform sketch-based image search by creating a correlation equation, generating a plurality of sketches for each category, and then providing the sketches. Figure 5 is a flowchart illustrating a method of performing sketch-based image search by providing a sketch according to an embodiment of the present invention.

Referring to FIG. 5, the user may request a sketch for sketch-based image search through manipulation of the user device 200. In this way, when the user device 200 receives a request for a sketch, the user context is extracted in step S210 and entered into the previously created correlation equation to derive a category, and a plurality of sketches mapped to the derived categories are displayed on the screen in order of priority. Displayed in thumbnail format. At this time, when the user selects one of the plurality of displayed sketches, the user enlarges the selected sketch and displays it.

The user can refer to the selected draft and make a touch input on the draft. Then, the user device 200 simultaneously displays the sketch and the touch input in step S220 and generates a trace of the touch input as a sketch. When the sketch is completed, the user device 200 transmits the selected draft and sketch to the search server 100 in step S230.

The search server 100, which has received the draft and sketch, performs an image search using the sketch as an image search term in step S240, and performs the image search by analyzing the differences between the draft and the sketch. And when the image search is completed in step S250, the search server 100 provides search results including a plurality of images to the user device 200. Then, the user device 200 displays a search result including a plurality of images received in step S260.

Above, embodiments of the present invention have been briefly described. Then, each of the above-described steps will be described in more detail.

First, the data collection method in step S110 will be described in more detail. Figure 6 is a flowchart for explaining a data collection method according to an embodiment of the present invention. Figure 7 is a diagram for explaining a data collection method according to an embodiment of the present invention.

The user can perform an image search by manipulating the user device 200. Each time such an image search is performed, the sketch providing module 400 of the control unit 290 of the user device 200 detects an image search event in step S310, and before a predetermined period of time based on the image search time point (Ts) in step S320 ( A plurality of user contexts are collected during the period (T1) from Tb) to the image search time (Ts). Through this, by collecting the context of a predetermined period (T1) before the search, it is possible to estimate what environment was created before the image search was performed and whether the image search was performed in this environment. For example, a user may have a behavior pattern of searching for images at a cafe near his or her workplace during lunch hours. Additionally, a user may have a behavior pattern of performing an image search while using an internet shopping app. For this reason, the sketch providing module 400 collects the context of a predetermined period (T1) before the search.

Next, the sketch providing module 400 detects at least one valid image during a predetermined period (T2) after a predetermined time (Tp) after the image search time (Ts) in step S320 and stores it in the storage unit 280. When a search is performed, a plurality of images are provided as search results, and when the user performs a follow-up operation such as clicking to check or downloading among the plurality of images belonging to the search results, the sketch providing module 400 provides these images. The image can be detected and saved as a valid image.

Then, the sketch providing module 400 determines the category of the valid image detected in step S330, maps it to the collected user context and the detected valid image, and stores it in the storage unit 280.

Next, as described above, after data collection is completed, the method for generating the correlation equation in step S120 will be described in more detail. Figure 8 is a flowchart illustrating a method for generating a correlation equation according to an embodiment of the present invention.

Referring to FIG. 8, the sketch providing module 400 selects a plurality of user contexts whose correlation with the image search event is a predetermined value or more from among the plurality of user contexts in step S410. For example, the sketch providing module 400 calculates the correlation coefficient (v) indicating the degree of correlation with the image search event for each of the plurality of user contexts as shown in Equation 1 below, and determines the correlation coefficient (v) of the plurality of user contexts whose correlation coefficient is more than a predetermined value. can be selected as a candidate user context.

Here, R is the image search frequency per unit period, and X represents the collected user context value. Additionally, cov(R, X) is the covariance of R and As such, according to an embodiment of the present invention, whenever an image search event occurs, a user context in which the user context value changes significantly and linearly is selected.

Next, the sketch providing module 400 derives a plurality of user contexts whose correlation with each other is more than a predetermined value among the plurality of user contexts selected in step S420, and the correlation with the image search event among the derived plurality of user contexts Select the single user context with the highest value. That is, one user context with the highest correlation with the image search event is selected among a plurality of user contexts whose correlations are above a predetermined value.

For example, the sketch providing module 400 can obtain a correlation coefficient (v) indicating the degree of correlation between a plurality of user contexts as shown in Equation 2 below.

where i and j are the indices of the user context, .

Additionally, the sketch providing module 400 may select the user context with the highest correlation coefficient with the previous image event among a plurality of user contexts whose mutual correlation coefficients are greater than a predetermined value.

For example, assume that the first user context is the commuting and leaving time, and the second user context is the subway, which is a means of transportation. Additionally, it is assumed that the first user takes the subway during rush hour and has a behavior pattern of performing image search only during rush hour. Additionally, it is assumed that the second user rides the subway during rush hour and performs image search even during rush hour, but always performs image search when he or she boards the subway at any time other than rush hour.

Each of the first user's first and second user contexts is highly correlated with image search. Since the first user takes the subway during rush hour, the correlation between both the first and second user contexts is also high. And among the first and second user contexts, the first user context will be ultimately selected because the commute and return times have a higher correlation with the image search compared to the speed of the user device 200 compared to the subway, which is a means of transportation. Each of the first and second user contexts of the second user is highly correlated with the image search. However, the correlation between the first and second user contexts is low because the second user always performs an image search when boarding the subway not only during rush hour but also at any time outside of rush hour. Therefore in this case both the first and second user contexts will be selected independently.

As described above, when the user context is selected, the sketch providing module 400 generates a correlation equation through linear regression analysis for the selected user context and category in step S430.

The method for generating the correlation equation is explained as follows. When k user contexts are selected, the linear regression equation is as follows:

Here, C is the probability of a category being selected in image search, and i is the index of the category. K represents the number of selected user contexts, and X1, X2, X3, ..., Xk represent different user contexts, respectively. Additionally, a0, a1, a2, a3, ..., an represent weights.

The sketch providing module 400 derives the weights a0, a1, a2, a3, ..., ak through linear regression analysis. That is, the sketch providing module 400 uses the values of the collected categories and the values of the contexts X1, X2, Multiple equations can be derived by entering them in 3). At this time, when a sufficiently large number of equations are derived by inputting a sufficiently large number of training data, the sketch providing module 400 combines a sufficiently large number of equations to obtain the weights a0, a1, a2, a3, as solutions to the simultaneous equations. .., ak can be calculated. Accordingly, the sketch providing module 400 can determine the linear regression equation from which the weights were calculated as a correlation equation. For example, if C1 represents the category women's clothing, k is 4, and the values of a0, a1, a2, a3, and a4 are 0.002, 0.208, 0.313, 0.025, and 0.472, the correlation equation is as follows:

According to this correlation equation, the correlation between the user context and the category of valid images selected by the user when searching for images is defined.

Next, we will explain how to create a sketch for each category in step S130. Figure 9 is a flowchart illustrating a method of generating a sketch for each category according to an embodiment of the present invention. 10 and 11 are diagrams for explaining a method of generating a sketch for each category according to an embodiment of the present invention.

Referring to FIG. 9, in step S510, the sketch providing module 400 groups a plurality of valid images collected for each category (S110) into a plurality of types according to shape characteristics to create a valid image group. Valid images for each category may be classified differently. For example, according to the search results for dresses in the category 'women's clothing', the dresses are Full Skirt Dress, A-line Skirt Dress, Wrap Dress, Empire Dress, Sheath Dress, BodyCon Dress, Baby Doll Dress, Shift Dress, and Spaghetti Straps. , Halter, etc. Therefore, in step S510, the sketch providing module 400 groups the valid images selected by the user according to the characteristics of the shape.

Then, in step S520, the sketch providing module 400 detects the outline of each of the plurality of valid images belonging to the valid image group according to a morphology analysis technique and generates a plurality of binary images consisting of only dots and lines. do. For example, as shown in FIG. 9, the valid image may be a photographic image, as shown in (a). Therefore, the outline of the effective image is derived and converted into a binary image consisting of only points and lines, as shown in (b).

Next, the draft providing module 400 overlaps a plurality of binary images and generates a draft according to the degree to which points and lines overlap. For example, as shown in (c) of FIG. 11, a plurality of binary images are overlapped and points or lines with a large degree of overlap are selected to create a sketch, as shown in (d).

Then, the draft providing module 400 determines the priority of the draft according to the number of valid images belonging to the valid image group. For example, if there are 3 valid images belonging to valid image group A and 20 valid images belonging to valid image group B, the draft generated from valid image group B has higher priority than the draft derived from valid image group A.

Above, we explained how to derive the correlation equation and create a sketch. In this way, we will explain how to perform sketch-based image search by providing the sketch while the correlation equation and sketch have been created.

First, we will explain in more detail how to display the sketch in step S210. Figure 12 is a flowchart for explaining a method of displaying a sketch according to an embodiment of the present invention. Figure 13 is an example screen for explaining a method of displaying a draft according to an embodiment of the present invention.

The user may request a sketch by manipulating the user device 200 to search for a sketch-based image. Accordingly, when the sketch providing module 400 of the user device 200 receives a request for a draft in step S610, it extracts the user context for a period from a predetermined period of time before the request to the time of the request in step S620.

Then, the sketch providing module 400 derives a category by inputting the value of the extracted context into a correlation equation such as Equation 4 in step S630.

When a category is derived, the sketch providing module 400 extracts the category derived in step S640 and a plurality of drafts belonging to the derived categories, and then arranges them in order of priority as shown in FIG. 13 to create the draft area (U). It is displayed in

Next, we will describe in more detail how to create a sketch in steps S220 and S230 and transmit the sketch as an image search term. Figure 14 is a flowchart illustrating a method of creating a sketch and transmitting the sketch as an image search word according to an embodiment of the present invention. Figure 15 is an example screen for explaining a method of creating a sketch according to an embodiment of the present invention.

Referring to Figures 14 and 15, the user can select one of a plurality of drafts in the draft area (U). Then, the sketch search module 500 of the user device 200 receives the user's selection in step S710 and displays the sketch selected by the user in the sketch area (K) of the screen.

Additionally, the user can refer to the sketch (ud) displayed in the sketch area (K) and make a touch input to the sketch area (K) using a finger or a stylus (ST).

Then, the sketch search module 500 detects a touch input on the sketch area (K) in step S720 and displays a trace of the touch input along with a sketch (ud) according to the detected touch input. This trace of touch input becomes a sketch (sd).

Once the sketch is completed, the user can request an image search for the sketch using a search term. If there is a request for such a sketch-based image search, the sketch search module 500 transmits the sketch (kd) and sketch (ud) to the search server 100 through the communication unit 210 in step S730.

Next, the sketch-based image search method in step S230 will be described. Figure 16 is a flowchart for explaining a sketch-based image search method according to an embodiment of the present invention. Figure 17 is a diagram for explaining a sketch-based image search method according to an embodiment of the present invention.

16 and 17, the image search module 300 of the control module 130 of the search server 100 receives a sketch (kd) and a draft (ud) from the user device 200 through the communication module 110. When received, the difference between the sketch (kd) and the draft (ud) is derived as a feature image (fd) in step S810. As shown in FIG. 17, the morphological difference between the sketch (kd) and the draft (ud) can be derived as a feature image (fd).

Next, the image search module 300 searches for a plurality of images (rd) that are identical or similar to the sketch (kd) in step S820. Then, the image search module 300 generates a search result in which the plurality of images (rd) searched in step S830 are sorted in order of high similarity to the feature image (fd). Compared to the sketch (ud), the characteristic image shows that there is no change in the sleeveless area, but the width of the skirt has been widened. Therefore, an image with a somewhat wide skirt can be determined to have a high degree of similarity.

The image search module 300 transmits to the user device 200 a search result in which the plurality of images searched through the communication module 110 are sorted in descending order of similarity to the feature image in step S840. Accordingly, the sketch search module 500 of the user device 200 may receive search results through the communication unit 210 and display the received search results through the display unit 260.

As described above, although this specification contains details of numerous specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather may be specific to particular embodiments of a particular invention. It should be understood as a description of features. Certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. Furthermore, although features may be described as operating in a particular combination and initially claimed as such, one or more features from a claimed combination may in some cases be excluded from that combination, and the claimed combination may be a sub-combination. It can be changed to a variant of a sub-combination.

Likewise, although operations are depicted in the drawings in a particular order, this should not be construed as requiring that those operations be performed in the specific order or sequential order shown or that all of the depicted operations must be performed to obtain desirable results. In certain cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. You must understand that it is possible.

Specific embodiments of the subject matter described herein have been described. Other embodiments are within the scope of the following claims. For example, the operations recited in the claims can be performed in a different order and still achieve desirable results. As an example, the process depicted in the accompanying drawings does not necessarily require the specific depicted order or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

The present description sets forth the best mode of the invention and provides examples to illustrate the invention and to enable any person skilled in the art to make or use the invention. The specification prepared in this way does not limit the present invention to the specific terms presented. Accordingly, although the present invention has been described in detail with reference to the above-described examples, those skilled in the art may make modifications, changes, and variations to the examples without departing from the scope of the present invention.

Therefore, the scope of the present invention should not be determined by the described embodiments, but by the scope of the patent claims.

The present invention relates to a method and device for sketch-based image retrieval. According to the present invention, when providing a sketch to help a user sketch, the user's current status is provided through analysis of user context at the time of image retrieval, which is a past case. We can provide the sketch you need. Accordingly, the user does not have to inconveniently search for numerous sketches and can conveniently perform a sketch-based search by selecting one of the plurality of sketches provided. Therefore, the present invention not only has sufficient marketability or commercial potential, but also has industrial applicability because it can be clearly implemented in reality.

100: Search server 110: Communication module
120: storage module 130: control module
200: User device 210: Communication department
220: Location information receiver 230: Sensor unit
240: camera unit 250: input unit
260: display unit 270: audio unit
280: storage unit 290: control unit
291: Clock module 300: Image search module
400: Sketch provision module 500: Sketch search module

Claims (13)

delete delete delete delete delete delete delete delete A communication unit that connects to a network and performs communication;
a display unit for displaying a screen including a sketch area and a draft area;
When a sketch-based image search is requested, the user context from a predetermined period of time prior to the request is extracted, and the extracted user context is entered into a correlation equation derived according to the correlation between the user context and the category of the previously performed image search results. a sketch providing module that derives a category through input, arranges the derived category and a plurality of drafts belonging to the derived category in order of priority, and displays them in the draft area; and
When one of the plurality of sketches is selected, the sketch is displayed in the sketch area of the screen, and when a touch input on the sketch area is detected, a trace of the touch input is displayed along with the sketch according to the detected touch input, A sketch-based image search device comprising a sketch search module that transmits the sketch and the sketch to a search server through the communication unit. According to clause 9,
The sketch providing module is
Whenever an image search is performed,
Collect a plurality of user contexts from a predetermined period of time before the image search time to the image search time, detect at least one valid image during a predetermined period after the image search time, and category of the detected valid image. Decide,
Deriving a user context whose correlation coefficient with the category is greater than or equal to a predetermined value among the plurality of collected user contexts,
Among the plurality of collected user contexts, select the user context in which the correlation coefficient between two or more user contexts is greater than a predetermined value and the correlation coefficient with the category is the highest,
A device for sketch-based image retrieval, characterized in that a correlation equation is generated through linear regression analysis of the selected user context and the category. According to clause 10,
The sketch providing module is
C is the probability that a category is selected, i is the index of the category, K is the number of selected user contexts, X1, When a3, ..., an are weights,
math equation to
Each time an image search is performed, the collected category and context values are input as training data to construct a plurality of simultaneous equations,
After calculating the weights as solutions of the constructed simultaneous equations,
A device for sketch-based image search, characterized in that it generates a correlation equation by applying the weight calculated to the above equation. According to clause 10,
The sketch providing module is
Create a valid image group by grouping multiple valid images collected for each category,
Detecting the outline of each of the plurality of valid images belonging to the valid image group according to a morphology analysis technique to generate a plurality of binary images consisting only of dots and lines,
Overlapping the plurality of binary images to generate a sketch according to the intensity of the overlapped points and lines,
A device for sketch-based image search, characterized in that the priority of the generated sketch is determined according to the number of a plurality of valid images belonging to the valid image group. delete

Images