KR20200023678A - A method for image searching using part of an image and an apparatus therefor - Google Patents

Abstract

The present invention relates to an image retrieval method using a part of an image and an apparatus therefor. The image retrieval method using a part of an image comprises the steps of: receiving, by an image processing module, an unfinished image of which a part is hided; generating, by the image processing module, a completed image by restoring the hided part through an artificial neural network; and performing, by the image processing module, image retrieval by using the completed image.

Description

A method for image searching using part of an image and an apparatus therefor}

The present invention relates to an image retrieval technology, and more particularly, to a method and an apparatus for retrieving an image using a partial image when a part of a search target object is hidden when performing an image retrieval using an image captured by a user. It is about.

Image search can be divided into two types according to the method and purpose of finding an image. The first is to search for related images using text search terms. For example, if you enter the keyword 'flowers', you can see different types of flowers. This type is a traditional search method from the very beginning of the Internet. The second is to enter an image search term, which is an image file, and analyze the image file to find the same or similar image. It's the way computer analysis and internet technology have developed. Compared to the keyword method, the user can intuitively search for an image.

Korean Laid-Open Patent No. 2010-0043823, published on April 29, 2010 (Name: Method for providing an online shopping function using a product image and a shopping mall server)

SUMMARY OF THE INVENTION An object of the present invention is to provide an image retrieval method and apparatus for an object partially hidden by another object among a plurality of objects when a plurality of objects are included in an image photographed by a user.

The image retrieval method according to a preferred embodiment of the present invention for achieving the object as described above comprises the steps of the image processing module to receive an unfinished image that is part of the image, the image processing module of the image through the artificial neural network Generating a partially reconstructed complete image, and performing an image search by the image retrieval module using the completed image.

The generating of the completed image may include inputting the unfinished image into the artificial neural network, and generating and outputting the finished image through a plurality of operations that are weighted with respect to the unfinished image through the artificial neural network. do.

Prior to generating the complete image, the image processing module further comprises the step of learning the artificial neural network so that the artificial neural network derives the complete image from the unfinished image.

The training of the artificial neural network may include: training the reconstructed network and the verification network of the artificial neural network individually by the image processing module; and training the reconstructed network and the verification network by the image processing module. Include.

The individually training may include inputting an incomplete image of an original image into the restoration network by the image processing module, and performing a plurality of operations in which weights are applied to the incomplete image into which the restoration network is input. Outputting the image processing module and modifying a weight of the restoration network such that a difference between the completed image generated by the restoration network and the original image is minimized.

Setting, by the image processing module, the verification network to determine that the completed image is a fake image, and determining a target value to determine that the original image is real, and wherein the image processing module determines the completed image or the original image on the verification network. Inputting to the outputting unit, and outputting, by the verification network, a probability that the input image or the probability that the input image is a real image or a fake image through an operation to which a plurality of weights are applied is output; And modifying a weight of the verification network so that a loss value that is a difference between an output value of the target value and the target value is minimized.

The training may be performed by setting a target value to determine that the completed image generated by the restoration network is fake, and then modifying the weight of the verification network so that the difference between the set target value and the output value of the verification network is minimized. And setting a target value to determine that the completed image generated by the restoration network is real, and then modifying the weight of the restoration network so that the difference between the set target value and the output value of the verification network is minimized. Include.

An image retrieval apparatus according to a preferred embodiment of the present invention for achieving the object as described above, if the communication module for communication with the user device, and an unfinished image is hidden part of the image is input, the image retrieval part An artificial neural network for outputting a reconstructed complete image, an image processing module for deriving the completed image through the artificial neural network upon receiving the unfinished image from the user device, and an image for performing image retrieval using the completed image Contains a search module.

When the unfinished image is input, the artificial neural network performs a plurality of operations in which weights are applied to the input unfinished image and outputs a completed image, and when the finished image or the original image is input, a plurality of weights are applied. It includes a verification network that outputs the probability that the completed image or the original image input through the operation is true or false.

The image processing module competes to learn by competing the reconstruction network and the verification network after performing individual learning to individually learn each of the reconstruction network and the verification network so that the reconstruction network generates a complete image from an incomplete image. Characterized in performing learning.

The image processing module inputs an incomplete image of the original image to the restoration network during the individual learning, and outputs a completed image by performing a plurality of operations to which a weight is applied to the incomplete image inputted by the restoration network. The weight of the restoration network is modified so that the difference between the completed image and the original image is minimized.

The image processing module sets the target value so that the image processing module determines that the verification image is a fake image and that the original image is real, and the image processing module determines that the original image is real. Inputting a complete image or an original image to the verification network, and outputting the probability that the input image or the original image is genuine and a fake value as an output value through an operation to which a plurality of weights are applied, the verification network The weight of the verification network is modified so that a loss value that is a difference between an output value and a target value is minimized.

The image processing module sets a target value to discriminate that the completed image generated by the reconstruction network is fake during the competitive learning, and then sets the minimum value between the set target value and the output value of the verification network. Modifying a weight, setting a target value to determine that the completed image generated by the restoration network is real, and then modifying the weight of the restoration network so that the difference between the set target value and the output value of the verification network is minimized. It features.

According to the present invention, an image retrieval may be performed using the completed image by completing the hidden portion of the hidden unfinished image of the object partially hidden. Accordingly, the accuracy of image retrieval can be improved.

1 is a view for explaining an image retrieval system according to an embodiment of the present invention.
2 and 3 are views for explaining an unfinished image, a finished image and an original image according to an embodiment of the present invention.
4 is a block diagram illustrating a configuration of a search server according to an embodiment of the present invention.
5 is a view for explaining the configuration of the artificial neural network according to an embodiment of the present invention.
6 is a view for explaining the configuration of the restoration network of the artificial neural network according to an embodiment of the present invention.
7 is a view for explaining the configuration of the verification network of the artificial neural network according to an embodiment of the present invention.
8 is a block diagram illustrating a configuration of a user device according to an exemplary embodiment of the present invention.
9 is a flowchart illustrating a method of learning an artificial neural network according to an embodiment of the present invention.
10 and 11 are flowcharts illustrating individual learning according to an embodiment of the present invention.
12 is a flowchart illustrating competitive learning according to an embodiment of the present invention.
13 is a flowchart illustrating an image search method according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS To make the features and advantages of the present invention more clear, the present invention will be described in more detail with reference to specific embodiments shown in the accompanying drawings.

However, in the following description and the accompanying drawings, detailed descriptions of well-known functions or configurations that may obscure the subject matter of the present invention will be omitted. In addition, it should be noted that like elements are denoted by the same reference numerals as much as possible throughout the drawings.

The terms or words used in the following description and drawings should not be construed as being limited to the common or dictionary meanings, and the inventors may appropriately define the concept of terms for describing their own invention in the best way. It should be interpreted as meanings and concepts corresponding to the technical idea of the present invention based on the principle that the present invention. Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical idea of the present invention, various modifications that can be substituted for them at the time of the present application It should be understood that there may be equivalents and variations.

In addition, terms including ordinal numbers, such as first and second, are used to describe various components, and are used only to distinguish one component from another component, and to limit the components. Not used. For example, without departing from the scope of the present invention, the second component may be referred to as the first component, and similarly, the first component may also be referred to as the second component.

In addition, when a component is referred to as being "connected" or "connected" to another component, it means that it may be connected or connected logically or physically. In other words, although a component may be directly connected or connected to other components, it should be understood that other components may exist in the middle and may be connected or connected indirectly.

In addition, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, the terms "comprises" or "having" described herein are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or the same. It is to be understood that the present invention does not exclude in advance the possibility of the presence or the addition of other features, numbers, steps, operations, components, parts, or a combination thereof.

In addition, the terms "?", "?", "Module", and the like described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. have.

In addition, "a" or "an", "one", "the" and the like shall also be indicated otherwise in the context of the present invention (particularly in the context of the following claims). Unless otherwise expressly contradicted by context, it may be used in the sense including both singular and plural.

In addition, embodiments within the scope of the present invention include computer readable media having or conveying computer executable instructions or data structures stored on computer readable media. Such computer readable media can be any available media that can be accessed by a general purpose or special purpose computer system. By way of example, such computer readable media may be in the form of RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or computer executable instructions, computer readable instructions or data structures. And may include, but are not limited to, any other medium that can be used to store or deliver certain program code means, and can be accessed by a general purpose or special purpose computer system. .

In the following description and claims, “network” is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules. When information is transmitted or provided to a computer system via a network or other (wired, wireless, or a combination of wired or wireless) communication connections, this connection can be understood as a computer-readable medium. Computer-readable instructions include, for example, instructions and data that cause a general purpose or special purpose computer system to perform a particular function or group of functions. The computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, or even source code.

In addition, the present invention relates to personal computers, laptop computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers In a network computing environment having various types of computer system configurations, including (pager) and the like, it may be applied to providing advertisements to the computer systems. The invention may also be practiced in distributed system environments where both local and remote computer systems that are linked via wired networks, wireless data links, or combinations of wired and wireless data links perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

First, an image retrieval system using an unfinished image according to an embodiment of the present invention will be described. 1 is a view for explaining an image retrieval system according to an embodiment of the present invention. 2 and 3 are diagrams for explaining an unfinished image, a completed image, and an original image according to an embodiment of the present invention. Referring to FIG. 1, an image search system includes a search server 100 and a user device 200.

The search server 100 and the user device 200 may transmit and receive data to and from each other through a communication network (NW). For example, the communication network NW may include a wireless communication network such as a wireless local area network (WLAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA). Depending on how the system is implemented, it may also include wired communication networks such as Ethernet, xDSL (ADSL, VDSL), Hybrid Fiber Coaxial Cable (HFC), Fiber to The Curb (FTTC), and Fiber To The Home (FTTH).

In addition, the communication network NW of the present invention may include, for example, a mobile communication network including a plurality of access networks (not shown) and a core network (not shown) connecting them. Here, the access network is a network that performs wireless communication by directly connecting to a terminal, for example, a plurality of base stations such as a base station (BS), a base transceiver station (BTS), a NodeB, an eNodeB, and a base station controller (BSC). It may be implemented as a base station controller such as a Radio Network Controller (RNC). In addition, as described above, the digital signal processing unit and the wireless signal processing unit, which are integrally implemented in the base station, are divided into digital units (hereinafter referred to as DUs and radio units (hereinafter referred to as RUs)). In addition, a plurality of RUs (not shown) may be installed in the area of the network, and a plurality of RUs may be connected to a centralized DU. It serves to connect networks, for example other communication networks, such as the Internet.

As described above, the core network is a network system that performs main functions for mobile communication services such as mobility control and switching between access networks, and performs circuit switching or packet switching, and performs mobile network. Manage and control the flow of packets within In addition, the core network may manage inter-frequency mobility and may play a role for interworking with traffic in the access network and the core network and other networks such as the Internet network. The core network may further include a serving gateway (SGW), a PDN gateway (PGW), a mobile switching center (MSC), a home location register (HLR), a mobile mobility entity (MME), and a home subscriber server (HSS). It may be.

In addition, the communication network (NW) according to the present invention may include an Internet network. The Internet network refers to a common public communication network, that is, a public network, through which information is exchanged according to the TCP / IP protocol. The search server 100 and the user device 200 interwork with each other through the communication network NW to form an image search system according to the present invention.

Here, the search server 100 is an entity existing on the network, and serves as a web server, a database server, and an application server. According to a preferred embodiment, the search server 100 searches for and provides an image identical or similar to a web server providing a shopping mall web site, a database server storing an image of an item sold on the web site, and an image search word. It can act as an application server. In the following embodiment, it is assumed that the search server 100 is a server that provides an Internet shopping mall service that sells various products. Accordingly, the search server 100 may search for images in the plurality of images stored in the database. Such an image sold in an internet shopping mall may be an image of a product. Meanwhile, according to another exemplary embodiment, the search server 100 may search for images from a plurality of web pages by accessing another server, that is, a web server, through a network.

In addition, the user device 200 will be described as a representative example of a mobile communication terminal, but the terminal is not limited to the mobile communication terminal, but a variety of information communication devices, multimedia terminals, wired terminals, fixed terminals and IP (Internet Protocol) terminals, etc. Applicable to the terminal. For example, the terminal may be a mobile terminal having various mobile communication specifications such as a mobile phone, a portable multimedia player (PMP), a mobile internet device (MID), a smart phone, a tablet PC, a tablet PC, and an information communication device. Can be advantageously used. As another example, the terminal may be a notebook, a personal computer, or the like.

As shown in FIG. 2, in the embodiment of the present invention, the image used as the image search word is an unfinished image I, and the unfinished image I is an image in which part of the object is covered by another object O. FIG. . Referring to FIG. 3, when a part of an object existing in the original image R is covered when the original image R exists, the image is referred to as an incomplete image I. FIG. In addition, the restoration of the hidden part of the unfinished image I will be referred to as the completed image C.

The user device 200 may request an image search by transmitting the unfinished image I as an image search word to the search server 100. Accordingly, when the search server 100 receives an image search word that is an incomplete image I, the search server 100 performs an image search for a search target object, that is, an object that is partially hidden. The search server 100 may complete the masked portion of the unfinished image I to generate a completed image C, and perform an image search using the generated completed image. Accordingly, the accuracy and reliability of image retrieval is improved.

Then, the configuration of the search server 100 and the user device 200 according to an embodiment of the present invention will be described in detail. First, the configuration of the search server 100 according to an embodiment of the present invention will be described. 4 is a block diagram illustrating a configuration of a search server according to an embodiment of the present invention. Referring to FIG. 4, the search server 100 includes a communication module 110, a storage module 120, and a control module 130.

The communication module 110 is for communication with the user device 200. The communication module 110 performs communication for exchanging data including necessary information with the user device 200 when the user device 200 accesses the search server 100. The communication module 110 extracts data from the packet received from the user device 200 and transfers the extracted data to the control module 130. For example, the extracted data may be an image search word. In addition, when the communication module 110 receives a search result including data for transmission from the control module 130 to the user device 200, the communication module 110 configures the received data into a packet and transmits the received data. For example, the received data may be a plurality of images that are image search results.

The storage module 120 stores a program and data necessary for the operation of the search server 100 and may be divided into a program area and a data area. The program area may store a program for controlling the overall operation of the search server 100, an operating system (OS) for booting the search server 100, an application for image processing, an application for image search, and the like. The data area is an area in which data generated according to the operation of the search server 100 and data necessary for operation of the search server 100 are stored. For example, when the search server 100 is an internet shopping mall, the data stored in the storage module 120 may include a product image of each product category. Such a product image may be an object of an image search. Various data stored in the storage module 120 may be deleted, changed, or added according to a user's manipulation.

The control module 130 may control a general operation of the search server 100 and a signal flow between internal blocks of the search server 100 and may perform a data processing function for processing data. The control module 130 is preferably a central processing unit (CPU). The control module 130 includes an artificial neural network 300, an image processing module 131, and an image retrieval module 133. When the unfinished image I is input, the artificial neural network 300 restores a part of the input unfinished image I to generate a completed image C. The image processing module 131 trains the artificial neural network 300 to generate the completed image C from the unfinished image I by using the plurality of training data. After the learning is completed, the image processing module 131 may receive an incomplete image from the user device 200 through the communication module 110. In this case, the image processing module 131 may derive the completed image from the unfinished image received through the artificial neural network 300. The image search module 133 receives the completed image from the image processing module 131 and performs an image search using the received completed image. Then, the image search module 133 transmits the searched image to the user device 200 through the communication module 110.

Then, the artificial neural network 300 according to an embodiment of the present invention will be described in more detail. 5 is a view for explaining the configuration of the artificial neural network according to an embodiment of the present invention. 6 is a view for explaining the configuration of the restoration network of the artificial neural network according to an embodiment of the present invention. 7 is a view for explaining the configuration of the verification network of the artificial neural network according to an embodiment of the present invention.

Referring to FIG. 5, the artificial neural network 300 according to the embodiment of the present invention includes a restoration network 310 and a verification network 320. Each of the restoration network 310 and the verification network 320 constitutes one artificial neural network.

When the incomplete image is input, the restoring network 310 outputs the completed image by performing a plurality of operations in which weights are applied to the input incomplete image. The verification network 320 is for verifying the completed image generated by the restoring network 310. When the completion image or the original image is input, the verification image 320 is a genuine input image through the plurality of calculations to which the weight is applied. Outputs probability and false probability as output values. Then, each of the restoration network 310 and the verification network 320 will be described.

The restoration network 310 will be described with reference to FIGS. 5 and 6. The recovery network 310 includes a plurality of layers including a plurality of operations to which weights are applied. Here, the plurality of layers including a plurality of operations may include a convolution layer (CL) for performing a convolution operation, a pooling layer (PL) for performing down sampling, and a pooling layer (PL) for performing a downsampling operation. At least one unpooling layer (UL) for performing an upsampling operation and one or more deconvolution layers (DL) for performing a deconvolution operation are included. Convolution, downsampling, upsampling, and deconvolution operations each use a filter of a predetermined matrix, with the values of the elements of those matrices being weighted. The restoration network 310 receives an unfinished image I. At this time, the pixel value of each pixel of the unfinished image I is input to the restoration network 310. When the unfinished image I is input, the reconstruction network 310 generates a complete image C by performing a plurality of operations in which weights of a plurality of layers are applied to each pixel value of the unfinished image I.

The verification network 320 will be described with reference to FIGS. 5 and 7. The verification network 320 also includes a plurality of layers that perform a plurality of operations to which weights are applied. Here, the plurality of layers that perform a plurality of operations may include a convolution layer (CL) that performs a convolution operation and a fully connected layer (FL) that performs a soft-max operation. Layer). The convolution operation uses a filter of a predetermined matrix, with the values of the elements of this matrix being weighted. Softmax operations are also performed by applying weights.

The verification network 320 is a fake representing the image restored by the restoration network 310 or not the input image is real or original image representing the original image is not generated by the restoration network 310 Outputs whether it is (fake). Here, the input image may be any one of the completed image C and the original image R generated by the restoration network 310.

Next, the user device 200 according to an embodiment of the present invention will be described. 8 is a block diagram illustrating a configuration of a user device according to an exemplary embodiment of the present invention. Referring to FIG. 8, the user device 200 according to an exemplary embodiment of the present invention may include a communication unit 210, a camera unit 220, an input unit 230, a display unit 240, a storage unit 250, and a controller 260. It includes.

The communication unit 210 is for communication with other devices including the search server 100 and performs communication through various communication connection methods. The communication unit 210 may communicate by selecting any one of various communication connection type communication functions. The communication unit 210 may be implemented by one module or a plurality of modules. For example, the communication unit 210 according to the WCDMA, LTE, LTE-A, etc. according to the broadband mobile communication method to access the network through the base station, according to the Wireless Local Area Network (WLAN) method using Wi-Fi (wireless fidelity) A wireless near field communication method for accessing a network NW through an access point (AP) and performing communication may be used. The communication unit 210 may include an RF transmitter for up-converting and amplifying a frequency of a transmitted signal, and an RF receiver for low noise amplifying and down-converting a received signal. Data received from the controller 260, for example, an image search word may be converted into a wireless signal and transmitted through a wireless channel. In addition, the communication unit 210 may receive a wireless signal including data, for example, an image search result, through a wireless channel, and transmit the data to the controller 260.

The camera unit 220 is for photographing an image. To this end, the camera unit 220 includes at least an image sensor. In the embodiment of the present invention, 'shooting' means generating an image generated by the image sensor of the camera unit 220 as a digital file according to a specific image file format (eg, JPG, PNG, etc.). The image sensor receives light reflected from a subject and converts the light into an electrical signal, and may be implemented based on a Charged Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS), or the like. The camera unit 220 may further include an analog-to-digital converter, and may convert the analog signal output from the image sensor into a digital signal and output the digital signal to the controller 260.

The input unit 230 receives a key operation of the user for controlling the user device 200, generates an input signal, and transmits the generated input signal to the controller 260. The input unit 230 may include any one of a power key, a numeric key, and a direction key for power on / off, and may be formed as a predetermined function key on one surface of the user device 200. When the display unit 240 is formed of a touch screen, functions of various keys of the input unit 230 may be performed by the display unit 240. When the display unit 240 may perform all functions using only the touch screen, the input unit 230 may be omitted. It may be.

The display unit 240 visually provides a menu, input data, function setting information, and various other information of the user device 200 to the user. In particular, the display unit 240 may display an image captured by the controller unit 260 through the camera unit 220. The display unit 240 outputs various screens such as a boot screen, a standby screen, a menu screen, and the like of the user device 200. The display unit 240 may be formed of a liquid crystal display (LCD), organic light emitting diodes (OLEDs), active matrix organic light emitting diodes (AMOLEDs), and the like. Meanwhile, the display unit 240 may be implemented as a touch screen. In this case, the display unit 240 may include a touch sensor, and the controller 260 may detect a user's touch input through the touch sensor. The touch sensor may be configured as a touch sensor such as capacitive overlay, pressure, resistive overlay, infrared beam, or a pressure sensor. . In addition to the above sensors, all kinds of sensor devices capable of detecting contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor detects a user's touch input, generates a detection signal, and transmits the detected signal to the controller 260. The sensing signal may include coordinate data input by the user. When the user inputs the touch position movement operation, the touch sensor may generate a detection signal including coordinate data of the touch position movement path and transmit the detected signal to the controller 260.

The storage unit 250 stores a program and data necessary for the operation of the user device 200, and may be divided into a program area and a data area. The program area may store a program for controlling the overall operation of the user device 200, an operating system (OS) for booting the user device 200, an application program, and the like. The data area is an area in which data generated according to the use of the user device 200 is stored. In addition, the storage 250 may store various data generated according to the operation of the user device 200.

The controller 260 may control a general operation of the user device 200 and a signal flow between internal blocks of the user device 200, and may perform a data processing function of processing data. The controller 260 may be a central processing unit (CPU), an application processor, a graphic processor (GPU), or the like.

The controller 260 may capture a specific image through the camera unit 220 according to a user's manipulation. The captured image may be an unfinished image (I). In addition, the controller 260 may access a specific web site through the communication unit 210 and download an image of the web site according to a user's manipulation. The downloaded image may be an incomplete image (I). The controller 260 selects an unfinished image I, which is an image taken or downloaded through the input unit 230 or the display unit 240, and detects an input requesting an image search for the selected image. The image search request including I) may be transmitted to the search server 100 through the communication unit 210 to perform an image search. The operation of this controller 260 will be described in more detail below.

As described above, the present invention restores the completed image (C) from the unfinished image (I) using the artificial neural network 300, and performs an image search using the completed image (C). To this end, first, the artificial neural network 300 must be trained so that the artificial neural network 300 restores the completed image C from the unfinished image I. First, this learning method will be described. 9 is a flowchart illustrating a method of learning an artificial neural network according to an embodiment of the present invention. 10 and 11 are flowcharts illustrating individual learning according to an embodiment of the present invention. 12 is a flowchart illustrating competitive learning according to an embodiment of the present invention.

Referring to FIG. 9, the image processing module 131 individually learns each of the reconstruction network 310 and the verification network 320 of the artificial neural network 300 in step S100. This learning will be referred to as 'individual learning'. The image processing module 131 trains the reconstruction network 310 to generate the completed image C from the unfinished image I during the individual learning. In addition, the image processing module 131 determines that the completed image C is fake, when the verification network 320 inputs the completed image C during individual learning, and the original image R is a fake image. Is inputted, the input original image R is trained to discriminate as being real.

In detail, the image processing module 131 performs individual learning on the reconstruction network 310 using the training data in step S110. In the individual learning, the image processing module 131 uses the original image R and the unfinished image I covering part of the original image R as the training data.

Referring to the individual learning of the reconstruction network 310 with reference to FIG. 10 in detail, the image processing module 131 inputs the unfinished image I, which is training data, to the reconstruction network 310 in step S111.

Then, the restoration network 310 performs a plurality of operations to which weights are applied to the unfinished image I input in step S113 and outputs the completed image C as an output value.

Then, the image processing module 131 compares the pixel values of the finished image (C) and the original image (R) with each other in step S115 and the loss value that is the difference between the finished image (C) and the original image (R) is the minimum. The weights applied to a plurality of operations of the restoration network 310 are modified through a back-propagation algorithm.

Next, the image processing module 131 performs initial learning on the verification network 320 in step S120. The image processing module 131 uses the completed image C and the original image R as learning data when the individual learning for the verification network 320 is performed.

Looking at the individual learning of the verification network 320 with reference to FIG. 11 in detail, the image processing module 131 determines that the verification network 320 is a fake image (C) in step S121, the original image (R) Set the target value to determine true). For example, the image processing module 131 sets a target value to real <fake, for example, 'real = 0.20' and 'fake = 0.80' when the completed image C is input. When the original image R is input, the target values may be set to real> fake, for example, 'real = 0.75' and 'fake = 0.25'.

The image processing module 131 inputs the learning data of any one of the completed image C and the original image R to the verification network 320 in step S123.

Then, the verification network 320 outputs, as an output value, the probability that the learning data input through the plurality of operations are real and fake in step S125. That is, when the completion image C or the original image R is input, the verification network 320 inputs the completed image C or the original image R through a calculation to which a plurality of weights are applied. Outputs the probability of a job and the probability of a fake. For example, the verification network 320 may output a probability of being real and a fake as an output value, such as 'real = 0.70' and 'fake = 0.30'.

Accordingly, the image processing module 131 is applied to a plurality of operations of the verification network 320 using a despreading algorithm such that the loss value, which is the difference between the output value of the verification network 320 and the target value, is minimized in step S127. Correct the weight.

Next, the image processing module 131 competes and learns the reconstruction network 310 and the verification network 320 of the artificial neural network 300 in step S200. This learning will be referred to as 'competitive learning'. At this time, the image processing module 131 sets a different target value for the verification network 320 and the restoring network 310 so as to learn the competition between the verification network 320 and the restoring network 310. That is, the image processing module 131 trains the verification network 320 so that the verification network 320 determines that the completed image C generated by the restoration network 310 is a fake. Correspondingly, the image processing module 131 trains the reconstruction network 310 so that the verification network 320 determines that the completed image C generated by the reconstruction network 310 is real.

Next, the competitive learning will be described in detail with reference to FIG. 12. First, the image processing module 131 sets the target value to discriminate the completed image C generated by the restoration network 310 as a fake and trains the verification network 320. To this end, the image processing module 131 sets a target value of the verification network 320 to determine that the completed image C generated by the restoration network 310 is a fake in step S210. For example, the image processing module 131 sets target values for the finished image C to real <fake, 'real = 0.15' and 'fake = 0.85'. In operation S220, the image processing module 131 inputs the unfinished image I, which is training data, to the restoration network 310. Then, the restoration network 310 generates a complete image (C) from the unfinished image (I) in step S230. Subsequently, the image processing module 131 inputs the completed image C obtained from the reconstruction network 310 to the verification network 320 in step S240. Then, the verification network 320 outputs the probability that the completed image C is real and a fake in step S250 as output values.

Accordingly, the image processing module 131 uses the despreading algorithm for the plurality of operations of the restoration network 310 to minimize the loss value that is the difference between the output value of the verification network 320 and the target value in operation S260. The weights of the plurality of operations of the verification network 320 are not modified.

As described above, in response to the learning of the verification network 320 in steps S210 to S260, the image processing module 131 generates the completed image C generated by the restoration network 310 by the verification network 320. The retrieval network 310 is trained by setting a target value to determine that it is. To this end, the image processing module 131 sets a target value so that the verification network 320 determines that the completed image C generated by the restoration network 310 is real in step S270. That is, the image processing module 131 may set a target value for the completed image C to real> fake, for example, 'real = 0.90' and 'fake = 0.10'.

Then, the image processing module 131 inputs the unfinished image I to the restoration network 310 in step S280. Then, the restoration network 310 generates a complete image (C) from the unfinished image (I) input in step S290. In operation S300, the image processing module 131 inputs the completed image C generated by the reconstruction network 310 to the verification network 320.

Then, the verification network 320 outputs the probability that the completed image C is real and the fake as an output value in step S310.

Accordingly, the image processing module 131 does not modify the weight of the verification network 320 by using the despreading algorithm so that the loss value that is the difference between the output value of the verification network 320 and the target value is minimized in step S320. The weight of the restoration network 310 is corrected. As described above, in the competitive learning, the learning about the restoration network 310 and the verification network 320 compete. These steps S210 to S320 are repeatedly performed.

In this repeated competitive learning process, the image processing module 131 determines whether the weights of the reconstruction network 310 and the verification network 320 change in step S330. If, as a result of the determination in step S330, if there is a change in the weight of at least one of the restoration network 310 and the verification network 320, steps S210 to S320 are repeated. As a result of the determination in step S340, if there is no change in the weight of both the restoration network 310 and the verification network 320, the competitive learning is terminated.

As described above, the derived image C may be derived from the unfinished image I by using the artificial neural network 300 on which learning is completed, and then image searching may be performed. Next, an image retrieval method according to an embodiment of the present invention will be described. 13 is a flowchart illustrating an image search method according to an embodiment of the present invention.

Referring to FIG. 13, the control unit 260 of the user device 200 generates an image search word and transmits the image search word generated through the communication unit 210 to the search server 100 in operation S410. Here, the unfinished image I may be an image captured by the control unit 260 through the camera unit 220 according to a user's operation, or may be an image downloaded from a corresponding web site by accessing a specific website through the communication unit 210. Can be. For example, as shown in FIG. 2, the unfinished image I is partially hidden by another object O. As shown in FIG.

When the image processing module 311 of the search server 100 receives the unfinished image I through the communication module 110, the completed image (C) from the unfinished image (I) using the artificial neural network 300 in step S420 ). That is, the image processing module 311 inputs the unfinished image I into the reconstruction network 310 of the artificial neural network 300 where the learning is completed. Then, the reconstruction network 310 of the artificial neural network 300 performs a plurality of operations by applying the learned weights to reconstruct a part of the unfinished image I to generate a completed image C. Accordingly, the complete image C is provided to the image search module 133.

The image search module 133 performs an image search on the completed image C as an image search word at step S430. According to an embodiment, the search server 100 is a server providing an internet shopping mall service for selling various products, and the database of the storage module 120 is assumed to be a plurality of images sold in the internet shopping mall. Then, the image search module 133 may perform an image search for the product image stored in the storage module 120 using the completed image C as an image search word. Meanwhile, according to another exemplary embodiment, the image search module 133 may search for images from a plurality of web pages by accessing another server on the network, that is, a web server through the communication module 110. Accordingly, the image search module 133 may derive a plurality of images identical or similar to the completed image C as the image search result. Then, the image search module 133 transmits an image search result including a plurality of images searched through the communication module 110 to the user device 200 in step S440.

When the user device 200 receives the image search result, the controller 260 of the user device 200 may display the plurality of images included in the image search result through the display unit 240 in operation S450.

As described above, the specification includes the details of a number of specific implementations, but these should not be understood as being limited to the scope of any invention or claimable, but rather may be specific to a particular embodiment of a particular invention. It should be understood as a description of the features. Certain features that are described in this specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, while the features operate in a specific combination and may be depicted as such initially claimed, one or more features from the claimed combination may in some cases be excluded from the combination, the claimed combination being a subcombination. Or a combination of subcombinations.

Likewise, although the operations are depicted in the drawings in a specific order, it should not be understood that such operations must be performed in the specific order or sequential order shown in order to obtain desirable results or that all illustrated operations must be performed. In certain cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that it can.

Specific embodiments of the subject matter described in this specification have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order but still achieve desirable results. As an example, the process depicted in the accompanying drawings does not necessarily require that particular illustrated or sequential order to obtain the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

The foregoing description presents the best mode of the invention, and provides examples to illustrate the invention and to enable those skilled in the art to make and use the invention. The specification thus produced is not intended to limit the invention to the specific terms presented. Thus, while the present invention has been described in detail with reference to the examples described above, those skilled in the art can make modifications, changes and variations to the examples without departing from the scope of the invention.

Therefore, the scope of the present invention should be determined by the claims rather than by the described embodiments.

The present invention relates to an image retrieval method using a portion of an image and an apparatus therefor, according to the present invention, by completing the hidden portion of the masked unfinished image of the object is partially hidden by performing an image search using the completed image, The accuracy and reliability of image retrieval can be improved. Therefore, the present invention has industrial applicability because the present invention is not only sufficiently commercially available or commercially viable, but also practically clearly implemented.

100: search server 110: communication module
120: storage module 130: control module
131: image processing module 133: image search module
200: user device 210: communication unit
220: camera unit 230: input unit
240: display unit 250: storage unit
260: control unit 300: artificial neural network
310: restoration network 320: verification network

Claims (13)

Receiving, by the image processing module, an incomplete image in which part of the image is covered;
Generating, by the image processing module, a completed image in which a part of the image is restored through an artificial neural network; And
And performing an image search by the image search module using the completed image. The method of claim 1,
Generating the finished image
Inputting the unfinished image into the artificial neural network;
And generating and outputting the finished image through a plurality of operations in which weights are applied to the unfinished image through the artificial neural network. The method of claim 1,
Before generating the finished image,
And training, by the image processing module, the artificial neural network so that the artificial neural network derives the finished image from the unfinished image. The method of claim 3,
Learning the artificial neural network
The image processing module individually learning each of the restoration network and the verification network of the artificial neural network; And
And training, by the image processing module, competing the reconstruction network and the verification network. The method of claim 4, wherein
The learning individually
Inputting, by the image processing module, an incomplete image of an original image into the restoration network;
Outputting a completed image by performing a plurality of operations to which a weight is applied to the unfinished image inputted by the restoration network;
And correcting, by the image processing module, a weight of the restoration network such that a difference between the completed image generated by the restoration network and the original image is minimized. The method of claim 4, wherein
Setting, by the image processing module, the verification network to determine that the completed image is a fake image, and to set a target value to determine that the original image is real;
Inputting, by the image processing module, a complete image or an original image into the verification network;
Outputting, by the verification network, a probability that the completed image or the original image that is input through a calculation to which a plurality of weights are applied is true and a probability of being false;
And correcting, by the image processing module, a weight of the verification network so that a loss value that is a difference between an output value of the verification network and a target value is minimized. The method of claim 4, wherein
Competing and learning
Setting a target value to determine that the completed image generated by the restoration network is a fake image, and then modifying a weight of the verification network so that a difference between the set target value and the output value of the verification network is minimized; And
And setting a target value to determine that the completed image generated by the restoration network is real, and correcting a weight of the restoration network such that a difference between the set target value and the output value of the verification network is minimized. An image search method characterized by the above-mentioned. Communication module for communication with the user device;
An artificial neural network for outputting a completed image in which a part of the image is masked and a partial image of the input unfinished image is restored;
An image processing module for deriving the completed image through the artificial neural network when receiving the unfinished image from the user device; And
And an image search module for performing an image search using the completed image. The method of claim 8,
The artificial neural network
A reconstruction network that outputs a complete image by performing a plurality of operations in which weights are applied to the input unfinished image when the incomplete image is input; And
A verification network outputting a probability that the input image or the probability that the input image or the original image is real and a fake value is output through a plurality of operations to which weights are applied when the completion image or the original image is input; Device. The method of claim 8,
The image processing module
The restoration network to generate a complete image from the unfinished image
After performing the individual learning to individually learn each of the restoration network and the verification network,
And an image retrieval device configured to compete by learning the reconstruction network and the verification network. The method of claim 10,
The image processing module
In the above individual study,
Inputting an unfinished image of the original image into the restoration network,
When the restoring network outputs the completed image by performing a plurality of operations to which the weight is applied to the input unfinished image, the weight of the restoring network is modified so that the difference between the output complete image and the original image is minimized. Image retrieval device. The method of claim 10,
The image processing module
In the above individual study,
The image processing module sets the target value so that the verification network determines that the completed image is fake, sets the target value to determine that the original image is real, inputs the completed image or the original image to the verification network,
If the verification network outputs the probability that the completed image or the original image is genuine and the fake value as an output value through an operation to which a plurality of weights are applied, the loss value that is the difference between the output value and the target value of the verification network is minimal And modifying the weight of the verification network so that it is correct. The method of claim 10,
The image processing module
In the competitive learning,
Setting a target value to determine that the completed image generated by the restoration network is fake, and then modifying a weight of the verification network so that a difference between the set target value and the output value of the verification network is minimized,
And setting a target value to determine that the completed image generated by the restoration network is real, and then modifying the weight of the restoration network so that the difference between the set target value and the output value of the verification network is minimized. Device.

Images