KR20200027586A - A method for image searching using a converted image and an apparatus therefor - Google Patents

Abstract

The present invention relates to an image search method using a part of an image and to an apparatus therefor. The present invention provides the image search method and the apparatus according thereto. The image search method comprises the following steps. An image processing module receives an image search word. The image processing module identifies a category of a query object by recognizing a query object included in the image search word. The image processing module generates a processed image which has processed a composition of a query object of the image search word to a composition same with a reference object included in a reference image corresponding to the category. The image search module performs an image search using the processed image.

Description

A method for image searching using a converted image and an apparatus therefor

The present invention relates to an image search technology, and more specifically, when performing an image search using an image captured by a user, after processing the image according to the composition of the image stored in the search target database, the processed image is processed. It relates to a method and apparatus for performing image search using.

Image search can be roughly divided into two types depending on the method and purpose of finding the image. The first is to find related images using text search terms. For example, if you enter the keyword 'flower' in the text, you can see various types of flower photos. This type is a traditional search method that has existed since the early days of the Internet. The second is to input an image search term, which is an image file, and analyze the image file to find the same or similar image. This is the way it emerged as computer analysis and Internet technology developed. It has the advantage that the user can intuitively search the image compared to the keyword method.

Published Korean Patent No. 2010-0043823, published on April 29, 2010 (Name: Method for providing online shopping function using product image and shopping mall server)

An object of the present invention is to provide a method and apparatus for performing an image search using a processed image that is converted to the composition of an image to be searched by converting the composition of an object of an image search term.

An image retrieval method according to a preferred embodiment of the present invention for achieving the above object includes the steps of an image processing module receiving an image search term, and the image processing module recognizing the query object included in the image search term to query the image. Identifying the category of the object, and the image processing module generates a processed image in which the composition of the query object of the image search term is processed through the artificial neural network to the same composition as the reference object included in the reference image corresponding to the identified category. And an image search module performing an image search using the processed image.

In the generating of the processed image, the image processing module inputs the image search word into the artificial neural network, and the image processing module performs a plurality of calculations in which weights are applied to the image search word through the artificial neural network. And generating and outputting the processed image.

Before the step of generating the processed image, the image processing module further includes learning the artificial neural network so that the artificial neural network generates the processed image from the image search word.

In the step of learning the artificial neural network, the image processing module separately learns each of the artificial neural network generation network and the authentication network, and the image processing module competes with the generation network and the authentication network for learning. Includes.

In the step of individually learning, the image processing module inputs a learning image into the generation network, and outputs a processed image by performing a plurality of calculations in which weights are applied to the learning image input by the generation network. And, the image processing module correcting a weight of the generation network so that a difference between the processed image generated by the generation network and the reference image is minimized.

Setting the target value so that the image processing module determines that the authentication network is fake, and sets the target value to determine the reference image as real; and the image processing module determines the processed image or reference image as the authentication network. Inputting the data, and outputting the probability that the processed image or the reference image that is input through a calculation in which the plurality of weights are applied is real and the probability of being fake as output values, and that the image processing module receives the authentication network. And modifying the weight of the authentication network so that the loss value, which is the difference between the output value and the target value of, is minimum.

The step of learning by competing is to set a target value to determine that the processed image generated by the generation network is fake, and then modify the weight of the authentication network so that the difference between the set target value and the output value of the authentication network is minimized. And setting a target value to determine that the processed image generated by the generation network is real, and then correcting the weight of the generation network so that the difference between the set target value and the output value of the authentication network is minimized. Includes.

An image retrieval apparatus according to a preferred embodiment of the present invention for achieving the above object is a communication module for receiving an image search word from a user device, and the composition of the query object of the image search word corresponds to the category of the query object The artificial neural network that generates a processed image processed with the same composition as the reference object included in the reference image, and the category of the item by recognizing the query object included in the image search term, and identifies the category of the item from the image search term through the artificial neural network. It includes an image processing module for generating a processed image, and an image search module for performing an image search using the processed image.

The artificial neural network includes a generation network that outputs a processed image by performing a plurality of calculations that apply weights to the input image search term when an image search term is input, and a plurality of weights to which a weight is applied when a processed image or a reference image is input. It includes an authentication network that outputs the probability that the processed image or the reference image input through the operation is real and fake.

The image processing module performs an individual learning to individually learn each of the generation network and the authentication network so that the generation network generates a processed image from an image search word, and then competes to learn by competing the generation network and the authentication network. Characterized by performing learning.

When the individual learning, the image processing module inputs an image search word of a reference image to the generation network and outputs a processed image by performing a plurality of calculations in which the generation network is weighted with respect to the input image search term. It is characterized in that the weight of the generation network is modified to minimize the difference between the processed image and the reference image.

In the individual learning, the image processing module sets a target value so that the image processing module determines that the authentication network is fake and the reference image is real, and then sets the processed image or the image. When a reference image is input to the authentication network, and when the processed image or a reference image input through a calculation in which a plurality of weights are applied, the authentication network outputs a probability of being real and a probability of being fake, as output values, the output value of the authentication network and It is characterized in that the weight of the authentication network is corrected so that a loss value, which is a difference between target values, is minimized.

The image processing module sets the target value to determine that the processed image generated by the generation network is fake during the competition learning, and then sets the target network so that the difference between the set target value and the output value of the authentication network is minimal. After modifying the weight and setting a target value to determine that the processed image generated by the generation network is real, modifying the weight of the generation network so that the difference between the set target value and the output value of the authentication network is minimum It is characterized by.

According to the present invention, after deriving a category to which a query object of an image search term belongs, a processed image processing a composition of a query object with the same composition as the reference object included in the reference image of the corresponding category is generated, and the generated processed image is used To perform image search. Accordingly, an image search may be performed through an image having the same composition as the image to be searched, thereby improving the accuracy and reliability of the image search.

1 is a view for explaining an image search system according to an embodiment of the present invention.
2 is a view for explaining a processed image according to an embodiment of the present invention.
3 is a block diagram illustrating the configuration of a search server according to an embodiment of the present invention.
4 is a view for explaining the configuration of an artificial neural network according to an embodiment of the present invention.
5 is a view for explaining the configuration of a generation network among artificial neural networks according to an embodiment of the present invention.
6 is a view for explaining the configuration of an authentication network among artificial neural networks according to an embodiment of the present invention.
7 is a block diagram illustrating the configuration of a user device according to an embodiment of the present invention.
8 is a flowchart illustrating a method of learning an artificial neural network according to an embodiment of the present invention.
9 and 10 are flowcharts for explaining individual learning according to an embodiment of the present invention.
11 is a flowchart illustrating a method of performing competitive learning according to an embodiment of the present invention.
12 is a flowchart illustrating an image search method according to an embodiment of the present invention.

In order to clarify the features and advantages of the problem solving means of the present invention, the present invention will be described in more detail with reference to specific embodiments of the present invention shown in the accompanying drawings.

However, in the following description and accompanying drawings, detailed descriptions of well-known functions or configurations that may obscure the subject matter of the present invention are omitted. In addition, it should be noted that the same components throughout the drawings are indicated by the same reference numerals as much as possible.

The terms and words used in the following description and drawings should not be interpreted as being limited to ordinary or dictionary meanings, and the inventor can appropriately define the concept of terms for explaining his or her invention in the best way. Based on the principle that it should be interpreted as meanings and concepts consistent with the technical spirit of the present invention. Therefore, the embodiments shown in the embodiments and drawings described in this specification are only the most preferred embodiments of the present invention, and do not represent all of the technical spirit of the present invention, and thus can replace them at the time of application. It should be understood that there may be equivalents and variations.

In addition, terms including ordinal numbers such as first and second are used to describe various components, and are used only to distinguish one component from other components, and to limit the components It is not used. For example, without departing from the scope of the present invention, the second component may be referred to as the first component, and similarly, the first component may also be referred to as the second component.

In addition, when referring to a component being "connected" or "connected" to another component, it means that it can be connected or connected logically or physically. In other words, it may be understood that a component may be directly connected to or connected to other components, but other components may exist in the middle and may be connected or connected indirectly.

In addition, the terms used herein are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, the terms "comprises" or "having" described herein are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or the same. It is to be understood that the present invention does not exclude in advance the possibility of the presence or the addition of other features, numbers, steps, operations, components, parts, or a combination thereof.

In addition, terms such as “… unit”, “… group”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. have.

In addition, "a (a or an)", "one (one)," "the (the)" and analogs are different in the context of describing the present invention (especially in the context of the following claims) as otherwise indicated herein. It may be used in a sense including both singular and plural unless it is or is clearly contradicted by the context.

In addition, embodiments within the scope of the present invention include computer-readable media having or having computer-executable instructions or data structures stored on computer-readable media. Such computer readable media can be any available media that can be accessed by a general purpose or special purpose computer system. By way of example, such computer readable media may be in the form of RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or computer executable instructions, computer readable instructions or data structures. Physical storage media, such as any other media that may be used to store or transfer certain program code means in, and can be accessed by, general purpose or special purpose computer systems. .

In the following description and claims, “network” is defined as one or more data links that enable electronic data to be transferred between computer systems and / or modules. When information is transmitted or provided to a computer system via a network or other (wired, wireless, or wired or wireless combination) communication connection, this connection may be understood as a computer-readable medium. Computer readable instructions include, for example, instructions and data that cause a general purpose computer system or special purpose computer system to perform a particular function or group of functions. The computer-executable instructions may be, for example, assembly language, or even binary, intermediate format instructions such as source code.

In addition, the present invention relates to personal computers, laptop computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers In a network computing environment having various types of computer system configurations, including (pager) and the like, it may be applied to providing advertisements to the computer systems. The invention can also be practiced in distributed system environments where both local and remote computer systems linked by wired data links, wireless data links, or combinations of wired and wireless data links over a network perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

First, an image search system using an image search word according to an embodiment of the present invention will be described. 1 is a view for explaining an image retrieval system according to an embodiment of the present invention. In addition, Figure 2 is a view for explaining a processed image according to an embodiment of the present invention. Referring to FIG. 1, an image search system includes a search server 100 and a user device 200.

The search server 100 and the user device 200 may transmit and receive data to and from each other through a communication network (NW). For example, the communication network (NW) may include a wireless communication network such as Wireless LAN (WLAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA). Depending on the system implementation method, it may include wired communication networks such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coaxial Cable), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home).

In addition, the communication network NW of the present invention may include, for example, a mobile communication network including a plurality of access networks (not shown) and a core network (not shown) connecting them. Here, the access network is a network that directly connects to a terminal and performs wireless communication, for example, a plurality of base stations such as a base station (BS), a base transceiver station (BTS), a NodeB, an eNodeB, and a base station controller (BSC). , RNC (Radio Network Controller) can be implemented as a base station controller. In addition, as described above, the digital signal processing unit and the radio signal processing unit, which are integrally implemented in the base station, are divided into digital units (hereinafter referred to as DU and radio units (hereinafter referred to as RU)), respectively. A plurality of RUs (not shown) may be installed in each area, and a plurality of RUs may be connected to a centralized DU to configure the core network (not shown) constituting the mobile network together with the access network. A network, for example, serves to connect other communication networks such as an Internet network.

As described above, the core network is a network system that performs a main function for mobile communication services such as mobility control and switching between access networks, and performs circuit switching or packet switching, and is a mobile network. Manage and control packet flow within. In addition, the core network may manage mobility between frequencies, and may play a role for interworking with traffic in the access network and the core network and other networks, such as the Internet network. Such a core network may further include SGW (Serving GateWay), PGW (PDN GateWay), MSC (Mobile Switching Center), HLR (Home Location Register), MME (Mobile Mobility Entity) and HSS (Home Subscriber Server). It might be.

In addition, the communication network (NW) according to the present invention may include an Internet network. The Internet network refers to a common public communication network, that is, a public network, through which information is exchanged according to the TCP / IP protocol. The search server 100 and the user device 200 interwork with each other through the communication network NW to form an image search system according to the present invention.

Here, the search server 100 is an entity existing on the network, and serves as a web server, a database server, and an application server. According to a preferred embodiment, the search server 100 searches for and provides an image identical or similar to a web server providing a shopping mall web site, a database server storing an image of an item sold on the web site, and an image search word. It can act as an application server. In the following embodiment, it is assumed that the search server 100 is a server that provides an Internet shopping mall service that sells various products. Accordingly, the search server 100 may search for images from a plurality of images stored in the database. Such an image sold in an internet shopping mall may be an image of a product. Meanwhile, according to another embodiment, the search server 100 may search for images from a plurality of web pages by accessing another server, that is, a web server, through a network.

In addition, the user device 200 will be described as a representative example of a mobile communication terminal, but the terminal is not limited to the mobile communication terminal, but a variety of information communication devices, multimedia terminals, wired terminals, fixed terminals and IP (Internet Protocol) terminals, etc. Applicable to the terminal. For example, the terminal may be a mobile terminal having various mobile communication specifications such as a mobile phone, a portable multimedia player (PMP), a mobile internet device (MID), a smart phone, a tablet PC, a tablet PC, and an information communication device. Can be advantageously used. As another example, the terminal may be a notebook computer or a personal computer.

As described above, the search server 100 according to an embodiment of the present invention may be a server providing an Internet shopping mall service. Accordingly, the search server 100 may store a plurality of product images including product objects in a database. 2, examples of such product images b1, b2, b3, b4, and b5 are shown. The product image including the product object stored in the database may be classified by product category. In addition, the product object of the product image according to each category has the same composition. As illustrated, the product images b1, b2, b3, b4, and b5 belong to category shoes and all have the same composition. According to an embodiment of the present invention, at least one reference image exists in one category, and the reference image includes a reference object having the same composition as the composition of the product object of the product image belonging to the category. For example, as shown in FIG. 2, a reference image s including a reference object is a reference image s corresponding to a category shoe, and the reference object (shoe) has the same composition as a product object belonging to the same category. Have

Meanwhile, the user device 200 may request an image search by transmitting an image search word to the search server 100. 2, examples of such image search terms a1, a2, a3, a4, and a5 are shown. As illustrated, the image search word may include a query object (oral shoe) to be searched for, and the composition of the query object varies even in the case of the same product. As shown in FIG. 2, the composition of the query object included in the image search term has the same composition as that of the product object of the same category of product images b1, b2, b3, b4, and b5, as in the image search term a5. Although it may be, it may have a composition different from that of the product object of the product images b1, b2, b3, b4, and b5 belonging to the same product category, such as the image objects a1, a2, a3, and a4. When the composition of the query object and the reference object is different compared to the same case, the precision and reliability of image search may be deteriorated. Accordingly, according to an embodiment of the present invention, when the search server 100 receives an image search word, derives a category to which the query object belongs, and processes the composition of the query object with the same composition as the reference object included in the reference image of the corresponding category One processed image is generated, and image search is performed using the generated processed image. Accordingly, accuracy and reliability of image retrieval are improved.

Then, the configuration of the search server 100 and the user device 200 according to an embodiment of the present invention will be described in detail. First, the configuration of the search server 100 according to an embodiment of the present invention will be described. 3 is a block diagram illustrating the configuration of a search server according to an embodiment of the present invention. Referring to FIG. 3, the search server 100 includes a communication module 110, a storage module 120, and a control module 130.

The communication module 110 is for communication with the user device 200. The communication module 110 performs communication for exchanging data including necessary information with the user device 200 when the user device 200 accesses the search server 100. The communication module 110 extracts data from the packet received from the user device 200 and transfers the extracted data to the control module 130. For example, the extracted data may be an image search word. In addition, when the communication module 110 receives a search result including data for transmitting to the user device 200 from the control module 130, the communication module 110 is configured to transmit the received data as a packet. For example, the received data may be a plurality of search images that are image search results.

The storage module 120 stores a program and data necessary for the operation of the search server 100 and may be divided into a program area and a data area. The program area may store a program for controlling the overall operation of the search server 100, an operating system (OS) for booting the search server 100, an application for image processing, an application for image search, and the like. The data area is an area in which data generated according to the operation of the search server 100 and data necessary for the operation of the search server 100 are stored. For example, when the search server 100 is an Internet shopping mall, data stored in the storage module 120 may include a product image for each product category. Such a product image may be an object of image search. Various types of data stored in the storage module 120 may be deleted, changed, or added according to user manipulation.

The control module 130 may control a general operation of the search server 100 and a signal flow between internal blocks of the search server 100 and may perform a data processing function for processing data. The control module 130 is preferably a central processing unit (CPU).

The control module 130 includes an artificial neural network 300, an image processing module 131, and an image search module 133.

When an image search word is input, the image processing module 131 identifies a category of a query object included in the image search word. In addition, the image processing module 131 generates a processed image by converting the composition of the query object of the image search term using the artificial neural network 300 so that the composition of the query object is the same as the reference object of the reference image corresponding to the identified category. do. To this end, the image processing module 131 trains the artificial neural network 300 to convert the composition of the query object to the same composition as that of the reference object using a plurality of training data.

After learning is completed, the image processing module 131 may receive an image search word from the user device 200 through the communication module 110. Accordingly, the image processing module 131 generates a processed image by converting the composition of the query object of the image search term to the same as that of the reference object through the artificial neural network 300.

The image search module 133 receives the processed image from the image processing module 131 and performs image search using the received processed image. That is, the image search module 133 performs image search to detect a plurality of search images identical or similar to the processed image. Then, the image search module 133 transmits a search result including a plurality of search images to the user device 200 through the communication module 110.

Then, the artificial neural network 300 according to an embodiment of the present invention will be described in more detail. 4 is a view for explaining the configuration of an artificial neural network according to an embodiment of the present invention. 5 is a view for explaining the configuration of a generation network among artificial neural networks according to an embodiment of the present invention. 6 is a view for explaining the configuration of an authentication network among artificial neural networks according to an embodiment of the present invention.

Referring to FIG. 4, the artificial neural network 300 according to an embodiment of the present invention includes a generating network 310 and an authentication network 320. Each of the generation network 310 and the authentication network 320 constitutes an artificial neural network.

When an image such as a learning image or an image search word is input, the generation network 310 generates a processed image by performing a plurality of calculations in which weights are applied to the input image. In other words, the generation network 310 is for generating a processed image from the input image when the image is input by simulating the reference image.

The authentication network 320 is an artificial neural network for assisting the learning of the generation network 310. Basically, the authentication network 320 distinguishes a processed image and a reference image. The authentication network 320 outputs the probability that the processed image or the reference image, which is input through a plurality of operations to which the weight is applied, is real and fake, as output values. When a processed image or a reference image is input, a probability that a processed image or a reference image inputted through a plurality of operations to which a weight is applied is real and a probability that it is a fake is output as output values. As described above, the generation network 310 creates a processed image by simulating a reference image. Accordingly, learning about the artificial neural network 300 according to an embodiment of the present invention trains the generation network 310 to the extent that the authentication network 320 determines the processed image generated by the generation network 310 as a reference image. That is done with the aim. Then, each of the generation network 310 and the authentication network 320 will be described.

First, the generation network 310 will be described with reference to FIGS. 4 and 6. The generation network 310 includes a plurality of layers including a plurality of operations to which weights are applied. Here, the plurality of layers including a plurality of operations includes a convolutional layer (CL) that performs a convolution operation, a pooling layer (PL) that performs a down sampling operation, and It includes at least one of an unpooling layer (UL) layer that performs an upsampling operation and a deconvolution layer (DL) that performs a deconvolution operation. Each of the convolution, downsampling, upsampling, and deconvolution operations uses a filter composed of a predetermined matrix, and the values of the elements of the matrix are weighted. The generation network 310 may receive any one of image search terms and learning images. At this time, pixel values of each pixel of the image are input to the generation network 310. The generation network 310 generates a processed image by performing a plurality of operations in which a weight of a plurality of layers is applied to each pixel value of the image when one of the image search word and the learning image is input. The processed image is an image processed with the composition of the object (learning object or query object) included in the input image (learning image or image search term) the same as the composition of the reference object of the reference image.

Next, the authentication network 320 will be described with reference to FIGS. 4 and 6. The authentication network 320 also includes a plurality of layers that perform a plurality of operations to which weights are applied. Here, the plurality of layers performing a plurality of operations is a convolutional layer (Convolution Layer) that performs a convolution operation (CL: Convolution Layer) and a soft-max (Soft-max) operation to perform a full connection layer (FL: Fully Connected) Layer). The convolution operation uses a filter composed of a predetermined matrix, and the values of the elements of the matrix are weighted. In addition, the softmax operation is also performed by applying weights. The image input to the authentication network 320 may be a processed image generated by the generation network 310 or a pre-stored reference image. The authentication network 320 generates a probability that the input image is a real image indicating that it is an original image that is not generated by the generation network 310, that is, a reference image, and an original image, that is, not a reference image. The probability of being a fake, which is a processed image generated by the network 310, is output.

Next, the user device 200 according to an embodiment of the present invention will be described. 7 is a block diagram illustrating the configuration of a user device according to an embodiment of the present invention. Referring to FIG. 7, the user device 200 according to an embodiment of the present invention includes a communication unit 210, a camera unit 220, an input unit 230, a display unit 240, a storage unit 250, and a control unit 260 It includes.

The communication unit 210 is for communication with other devices including the search server 100 and performs communication through various communication connection methods. The communication unit 210 may communicate by selecting any one of communication functions of various communication connection methods. The communication unit 210 may be implemented as a single module or a plurality of modules. For example, the communication unit 210 according to the WCDMA, LTE, LTE-A, etc. according to the broadband mobile communication method to access the network through the base station, according to the Wireless Local Area Network (WLAN) method using Wi-Fi (wireless fidelity) A wireless near field communication method for accessing a network NW through an access point (AP) and performing communication may be used. The communication unit 210 may be composed of an RF transmitter that up-converts and amplifies the frequency of the transmitted signal, and an RF receiver that amplifies the received signal with low noise and down-converts the frequency. Data received from the control unit 260, for example, an image search word may be converted into a wireless signal and transmitted through a wireless channel. In addition, the communication unit 210 may receive a wireless signal including data, for example, an image search result, through a wireless channel, and transmit the data to the controller 260.

The camera unit 220 is for photographing an image. To this end, the camera unit 220 includes at least an image sensor. In the embodiment of the present invention, 'shooting' means generating an image generated by the image sensor of the camera unit 220 as a digital file according to a specific image file format (eg, JPG, PNG, etc.). The image sensor receives light reflected from a subject and converts the light into an electrical signal, and may be implemented based on a Charged Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS), or the like. The camera unit 220 may further include an analog-to-digital converter, and may convert the analog signal output from the image sensor into a digital signal and output the digital signal to the controller 260.

The input unit 230 receives a key operation of the user for controlling the user device 200, generates an input signal, and transmits the generated input signal to the controller 260. The input unit 230 may include any one of a power key, a numeric key, and a direction key for power on / off, and may be formed as a predetermined function key on one surface of the user device 200. When the display unit 240 is made of a touch screen, functions of various types of keys of the input unit 230 may be performed on the display unit 240, and when all functions can be performed only by the touch screen, the input unit 230 may be omitted. It might be.

The display unit 240 visually provides a menu, input data, function setting information, and various other information of the user device 200 to the user. In particular, the display unit 240 may display an image captured by the controller unit 260 through the camera unit 220. The display unit 240 outputs various screens such as a boot screen, a standby screen, a menu screen, and the like of the user device 200. The display unit 240 may be formed of a liquid crystal display (LCD), organic light emitting diodes (OLEDs), active matrix organic light emitting diodes (AMOLEDs), and the like. Meanwhile, the display unit 240 may be implemented as a touch screen. In this case, the display unit 240 may include a touch sensor, and the controller 260 may detect a user's touch input through the touch sensor. The touch sensor may include a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, an infrared beam, or a pressure sensor. . In addition to the sensors, all kinds of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor detects a user's touch input, generates a detection signal, and transmits the detected signal to the controller 260. The sensing signal may include coordinate data input by the user. When the user inputs the touch position movement operation, the touch sensor may generate a detection signal including coordinate data of the touch position movement path and transmit the detected signal to the controller 260.

The storage unit 250 stores a program and data necessary for the operation of the user device 200, and may be divided into a program area and a data area. The program area may store a program for controlling the overall operation of the user device 200, an operating system (OS) for booting the user device 200, an application program, and the like. The data area is an area in which data generated according to the use of the user device 200 is stored. In addition, the storage 250 may store various data generated according to the operation of the user device 200.

The controller 260 may control a general operation of the user device 200 and a signal flow between internal blocks of the user device 200, and may perform a data processing function of processing data. The controller 260 may be a central processing unit (CPU), an application processor, a graphic processor (GPU), or the like.

The controller 260 may take a specific image through the camera unit 220 according to a user's manipulation. The captured image may be an image search term. Also, the controller 260 may access a specific website through the communication unit 210 and download an image of the website according to the user's operation. As such, the downloaded image may also be an image search term. The controller 260 selects an image search word that is an image or a downloaded image through the input unit 230 or the display unit 240 and detects an input requesting an image search for the selected image. The search request may be transmitted to the search server 100 through the communication unit 210 to perform image search. The operation of the control unit 260 will be described in more detail below.

As described above, the present invention generates a processed image from an image search word using the artificial neural network 300 and performs image search using the generated processed image. To this end, first, the artificial neural network 300 must be trained so that the artificial neural network 300 generates a processed image from an image search word. Then, first, this learning method will be described. 8 is a flowchart illustrating a method of learning an artificial neural network according to an embodiment of the present invention. 9 and 10 are flowcharts for explaining individual learning according to an embodiment of the present invention. 11 is a flowchart illustrating a method of performing competitive learning according to an embodiment of the present invention.

Referring to FIG. 8, the image processing module 131 separately performs learning on each of the generation network 310 and the authentication network 320 of the artificial neural network 300 in step S100. We will refer to this learning as 'individual learning'. When individually learning, the image processing module 131 trains the generation network 310 to generate a processed image from an image search word.

In addition, the image processing module 131, in the case of individual learning, when the authentication network 320 inputs a processed image, determines that the input processed image is a fake, and when a reference image is input, inputs the reference image. Train them to discriminate as real.

Specifically, the image processing module 131 performs individual learning on the generation network 310 using the learning data in step S110. The training data includes training images and reference images. The learning image is the same product as the reference object, but includes learning objects photographed in different compositions.

Referring to FIG. 9 in detail for the individual learning of the generation network 310, the image processing module 131 sets the reference image as a target value in step S111, and inputs the learning image to the generation network 310. Then, the generation network 310 will output a processed image as an output value by performing a plurality of operations to which a weight is applied to the learning image input in step S113. Then, the image processing module 131 compares the pixel values of the processed image and the reference image to each other in step S115, so that the difference between the pixel values of the processed image and the reference image is a back-propagation algorithm so that a loss value is minimized. By modifying the weight applied to a plurality of operations of the generation network 310 through.

Next, the image processing module 131 performs initial learning on the authentication network 320 in step S120. The image processing module 131 uses a processed image and a reference image as learning data when individually learning the authentication network 320.

Referring specifically to the individual learning of the authentication network 320 with reference to FIG. 10, the image processing module 131 determines in step S121 that the authentication network 320 identifies the processed image as a fake, and the reference image R is real. The target value is set to determine as being. For example, when the processed image is input, the image processing module 131 sets the target value to real <fake, for example, 'real = 0.20' and 'fake = 0.80', and the reference image When (R) is entered, the target value can be set to real> fake, such as 'real = 0.75' and 'fake = 0.25'.

In step S123, the image processing module 131 inputs learning data of one of the processed image and the reference image to the authentication network 320. Then, the authentication network 320 outputs the probability that the learning data input through a plurality of operations in the step S125 is real and the probability that it is a fake. That is, when the processed image or the reference image is input, the authentication network 320 outputs the probability that the processed image or the reference image input through a calculation to which a plurality of weights are applied is real and fake. Output. For example, the authentication network 320 may output a probability of being real and a probability of being fake as 'real = 0.70' and 'fake = 0.30'.

Accordingly, the image processing module 131 is applied to a plurality of calculations of the authentication network 320 using a despreading algorithm so that the loss value, which is the difference between the output value and the target value, of the authentication network 320 is minimum in step S127. Correct the weight.

Next, the image processing module 131 then competes and trains the generation network 310 and the authentication network 320 of the artificial neural network 300 in step S200. We will call this learning 'competitive learning'. At this time, the image processing module 131 performs learning to set the different target values for the authentication network 320 and the generation network 310 so that the authentication network 320 and the generation network 310 compete with each other. That is, the image processing module 131 trains the authentication network 320 so that the authentication network 320 determines that the processed image generated by the generation network 310 is a fake. In response to this, the image processing module 131 trains the generation network 310 so that the authentication network 320 determines that the processed image generated by the generation network 310 is real. That is, the image processing module 131 trains the generation network 310 to generate a processed image by simulating a reference image so that the authentication network 320 cannot recognize whether the processed image is fake.

Then, with reference to FIG. 11, competitive learning will be described in detail. First, the image processing module 131 trains the authentication network 320 by setting a target value to determine that the processed image generated by the generation network 310 is a fake. To this end, the image processing module 131 sets a target value of the authentication network 320 so as to determine that the processed image generated by the generation network 310 is a fake. For example, the image processing module 131 sets target values for the processed image as real <fake, 'real = 0.15' and 'fake = 0.85'. Then, the image processing module 131 inputs the learning image to the generation network 310 in step S220. Then, the generation network 310 generates a processed image from the learning image in step S230. Subsequently, the image processing module 131 inputs the processed image obtained from the generation network 310 to the authentication network 320 in step S240. Then, in step S250, the authentication network 320 outputs the probability that the processed image is real and the probability that it is fake as output values.

Accordingly, the image processing module 131 uses a despreading algorithm to minimize the loss value, which is the difference between the output value and the target value, of the authentication network 320 in step S260, for a plurality of operations of the generation network 310. The weights are not modified, and the weights for a plurality of operations of the authentication network 320 are corrected. That is, the image processing module 131 trains the authentication network 320 to determine that the processed image generated by the generation network 310 is a fake.

In response to the learning of the authentication network 320 in steps S210 to S260, the image processing module 131 determines that the processed image generated by the generation network 310 by the authentication network 320 is real. Set the target value so as to train the generation network (310). To this end, the image processing module 131 sets the target value so that the authentication network 320 determines in step S270 that the processed image generated by the generation network 310 is real. That is, the image processing module 131 may set a target value for the processed image as real> fake, for example, 'real = 0.90' and 'fake = 0.10'.

Then, the image processing module 131 inputs the learning image to the generation network 310 in step S280. Then, the generation network 310 generates a processed image from the learning image input in step S290. Subsequently, the image processing module 131 inputs the processed image generated by the generation network 310 in step S300 to the authentication network 320.

Then, then, the authentication network 320 outputs the probability that the processed image is real and the probability that it is fake as an output value in step S310.

Accordingly, the image processing module 131 does not modify the weight of the authentication network 320 using a despreading algorithm so that the loss value, which is the difference between the output value and the target value, of the authentication network 320 is minimized in step S320. , Modify the weight of the generation network 310. That is, the image processing module 131 trains the generation network 310 to generate a processed image by simulating a reference image so that the authentication network 320 cannot recognize whether the processed image is fake.

In this way, in the competition learning, the learning about the generation network 310 and the authentication network 320 compete. Steps S210 to S320 are repeatedly performed.

In the repeated competition learning process, the image processing module 131 determines whether the weights of the generation network 310 and the authentication network 320 are changed in step S330. If, as a result of the determination in step S330, there is a change in weight of at least one of the generation network 310 and the authentication network 320, steps S210 to S320 are repeated. As a result of the determination in step S340, if there is no change in the weight of both the generation network 310 and the authentication network 320, the competition learning ends.

As described above, learning of the artificial neural network 300 is performed to simulate a reference image of each of a plurality of categories. Accordingly, when the image search keyword is input, the artificial neural network 300 may generate a processed image that mimics a reference image corresponding to a category to which the image search keyword belongs. Accordingly, the image search may be performed using the artificial neural network 300 that has been completed. Then, an image search method according to an embodiment of the present invention will be described. 12 is a flowchart illustrating an image search method according to an embodiment of the present invention.

Referring to FIG. 12, the control unit 260 of the user device 200 generates an image search word in step S410. Here, the image search term may be an image captured by the control unit 260 through the camera unit 220 according to a user's manipulation, or an image downloaded from a corresponding website by accessing a specific website through the communication unit 210. . For example, as illustrated in FIG. 2, the query object of the image search word may be an image having a different composition from the reference object of the reference image. Next, the control unit 260 transmits the image search word generated through the communication unit 210 to the search server 100 in step S420.

When the image search module 311 of the search server 100 receives the image search word through the communication module 110, in step S430, the artificial neural network 300 is used to identify the category of the query object of the image search word. Then, in step S440, the image processing module 311 generates a processed image processed with the same composition as the reference object included in the reference image corresponding to the identified category through the artificial neural network 300 through the artificial intelligence network 300. do. That is, the image processing module 311 inputs the image search word into the generation network 310 of the artificial neural network 300 where learning is completed. Then, the generation network 310 of the artificial neural network 300 generates a processed image by performing a plurality of operations applying the learned weights. The generated processed image is provided to the image search module 133.

The image search module 133 performs an image search for the processed image in step S450. According to one embodiment, the search server 100 is a server that provides an Internet shopping mall service that sells various products, and it is assumed that a plurality of product images sold in the Internet shopping mall are stored in a database of the storage module 120. do. Then, the image search module 133 may perform an image search on the product image stored in the storage module 120 using the processed image as an image search term. Then, the image search module 133 transmits an image search result including a plurality of product images searched through the communication module 110 to the user device 200 in step S460.

When the user device 200 receives the image search result, the control unit 260 of the user device 200 may display a plurality of product images included in the image search result through the display unit 240 in step S450.

As described above, this specification includes details of a number of specific implementations, but these should not be understood as limiting with respect to any invention or claimable scope, but rather may be specific to a particular embodiment of a particular invention. It should be understood as a description of the features. Certain features that are described in this specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Further, although features may operate in a particular combination and may initially be depicted as so claimed, one or more features from the claimed combination may in some cases be excluded from the combination, and the claimed combination subcombined. Or sub-combinations.

Likewise, although the operations are depicted in the drawings in a particular order, it should not be understood that such operations should be performed in the particular order shown or in sequential order, or that all shown actions should be performed in order to obtain desirable results. In certain cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged in multiple software products. You should understand that you can.

Specific embodiments of the subject matter described herein have been described. Other embodiments are within the scope of the following claims. For example, the operations recited in the claims may be performed in different orders while still achieving desirable results. As an example, the process illustrated in the accompanying drawings does not necessarily require that particular illustrated order or sequential order to obtain desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

The described description presents the best mode of the present invention, and provides examples for explaining the present invention and for those skilled in the art to make and use the present invention. The specification thus produced is not intended to limit the invention to the specific terms presented. Therefore, although the present invention has been described in detail with reference to the above-described examples, those skilled in the art can make modifications, alterations, and modifications to these examples without departing from the scope of the present invention.

Therefore, the scope of the present invention should not be determined by the described embodiments, but should be determined by the claims.

The present invention relates to an image search method using a processed image and an apparatus therefor, according to the present invention, after deriving a category to which a query object of an image search term belongs, querying with the same composition as the reference object included in the reference image of the corresponding category A processed image processing the composition of the object is generated, and image search is performed using the generated processed image. Accordingly, it is possible to perform image search through an image having the same composition as the image to be searched, thereby improving the accuracy and reliability of the image search. Therefore, the present invention has industrial applicability because the present invention is not only sufficiently commercially available or commercially viable, but also practically clearly implemented.

100: search server 110: communication module
120: storage module 130: control module
131: image processing module 133: image search module
200: user device 210: communication unit
220: camera unit 230: input unit
240: display unit 250: storage unit
260: control unit 300: artificial neural network
310: generation network 320: authentication network

Claims (13)

An image processing module receiving an image search word;
An image processing module identifying a category of the query object by recognizing a query object included in the image search term;
Generating, by the image processing module, a processed image in which the composition of the query object of the image search term is processed to the same composition as the reference object included in the reference image corresponding to the identified category through an artificial neural network; And
And an image search module performing an image search using the processed image. According to claim 1,
The step of generating the processed image
Inputting the image search word into the artificial neural network by the image processing module;
And the image processing module generating and outputting the processed image through a plurality of calculations in which weights are applied to the image search terms through the artificial neural network. According to claim 1,
Before the step of generating the processed image,
And the image processing module learning the artificial neural network so that the artificial neural network generates the processed image from the image search word. The method of claim 3,
The step of learning the artificial neural network
The image processing module
Individually learning each of the artificial neural network generation network and the authentication network; And
And the image processing module competing and learning the generation network and the authentication network. According to claim 4,
The step of learning individually
Inputting a learning image into the generation network by the image processing module;
Outputting a processed image by performing a plurality of calculations in which a weight is applied to the learning image input by the generation network;
And modifying the weight of the generation network so that the difference between the processed image generated by the generation network and the reference image is minimized by the image processing module. According to claim 4,
Setting, by the image processing module, a target value such that the authentication network determines that the processed image is fake, and determines that the reference image is genuine;
Inputting a processed image or a reference image into the authentication network by the image processing module;
Outputting, as an output value, the probability that the authentication image is a real image or a reference image that is input through an operation in which the plurality of weights are applied by the authentication network;
And the image processing module correcting the weight of the authentication network so that a loss value, which is a difference between an output value and a target value of the authentication network, is minimum. According to claim 4,
The step of learning by competing is
Setting a target value to determine that the processed image generated by the generation network is fake, and then correcting the weight of the authentication network so that the difference between the set target value and the output value of the authentication network is minimal; And
And setting a target value to determine that the processed image generated by the generation network is real, and then correcting the weight of the generation network so that the difference between the set target value and the output value of the authentication network is minimum. Image search method characterized by. A communication module for receiving an image search word from a user device;
An artificial neural network generating a processed image in which the composition of the query object of the image search term is processed to the same composition as the reference object included in the reference image corresponding to the category of the query object;
An image processing module that recognizes a query object included in the image search term to identify the category of the item, and generates the processed image from the image search term through the artificial neural network; And
And an image search module for performing an image search using the processed image. The method of claim 8,
The artificial neural network
A generation network that outputs a processed image by performing a plurality of operations in which a weight is applied to the input image search word when the image search word is input; And
When the processed image or the reference image is input, the image search characterized in that it comprises; an authentication network that outputs the probability that the processed image or the reference image inputted through a plurality of operations to which the weight is applied is a real probability and a false probability as output values. Device. The method of claim 8,
The image processing module
After performing the individual learning to individually learn each of the generation network and the authentication network so that the generation network generates a processed image from the image search word,
An image retrieval apparatus characterized by performing a competitive learning to compete and learn the generation network and the authentication network. The method of claim 10,
The image processing module
In the above individual learning,
Enter an image search term of the reference image in the generation network,
When the generation network outputs a processed image by performing a plurality of calculations in which a weight is applied to the input image search word, the weight of the generation network is modified to minimize the difference between the output processed image and the reference image. Image search device. The method of claim 10,
The image processing module
In the above individual learning,
After the image processing module determines that the authentication network is fake, and sets a target value to determine the reference image as real, inputs the processed image or the reference image into the authentication network,
If the authentication network outputs the probability that the processed image or the reference image, which is input through a calculation in which multiple weights are applied, is a real probability and a false probability, as output values, the loss value, which is the difference between the output value and the target value of the authentication network, is the minimum value. The image retrieval device, characterized in that to modify the weight of the authentication network as possible. The method of claim 10,
The image processing module
In the competition learning,
After setting a target value to determine that the processed image generated by the generation network is fake, correct the weight of the authentication network so that the difference between the set target value and the output value of the authentication network is minimal,
After setting a target value to determine that the processed image generated by the generation network is genuine, an image search characterized by modifying the weight of the generation network to minimize the difference between the set target value and the output value of the authentication network Device.

Images