KR20200026422A - Apparatus and method for image search - Google Patents

Abstract

The present invention provides an image search method and an apparatus thereof. In the image search method, when a data collection module of a search server collects data including a video and an image from a web server, an image analysis module of the search server analyzes the collected data and determines whether the collected data is an image similar to a product image stored in a product image database. When determined as a similar image, a database update module of the search server adds the collected data to the product image database.

Description

Apparatus and method for image search

The present invention relates to an image retrieval technology, and more particularly, to an image retrieval apparatus and method for collecting and analyzing data including a video and an image from a web server to add data similar to the product image stored in the product image database will be.

Image search can be divided into two types according to the method and purpose of finding the image.

The first type is to find related images using text search terms. For example, if you enter the text search for "flowers," you can see different kinds of flower images. This type is a traditional search method from the early days of the Internet.

The second type is to enter the image file itself as a query image, and analyze the input image file to find the same or similar image. This type is a method that emerged with the development of computer analysis technology and the Internet technology, and has the advantage that the user can intuitively search the image compared to the type using the text search word.

Korean Laid-Open Patent No. 2010-0043823 (Name: Method for providing an online shopping function using a product image and a shopping mall server)

It is an object of the present invention to provide an image retrieval apparatus and method for adding data similar to a product image stored in a product image database by collecting and analyzing data including a video and an image from a web server.

The image retrieval method according to the present invention comprises the steps of: a data collection module collecting data including a video and an image from a web server; Analyzing, by the image analysis module, the collected data to determine whether the image is similar to a product image stored in a product image database; And if it is determined that the image is similar, the database updating module adds the collected data to the product image database.

If the collected data is an image, the determining may include: extracting an object from the collected image by the image analysis module, and the extracted object is similar to a product object of a product image stored in the product image database. The method may include determining whether the database update module is similar, and adding the collected image to the product image database by linking the collected image with the product object.

If the collected data is a video, the determining includes: extracting an object from each frame of the collected video by the image analysis module, and the extracted object is a product of a product image stored in the product image database. The method may include determining whether the object is similar to the object, and extracting a corresponding frame image of the collected video determined to be similar, and wherein the adding comprises: extracting the extracted frame image by the database update module. Linking to an object may include adding to the product image database.

If it is not determined that the image is similar, the image retrieval method may further include the image analysis module deleting the collected data or separately managing the dissimilar data.

An image retrieval apparatus according to the present invention includes a communication module for communicating with a plurality of web servers; A product image database for storing a plurality of product images; A data collection module for collecting data including video and images from the web servers through the communication module; An image analysis module for analyzing the collected data and determining whether the image is similar to a product image stored in the product image database; And a database update module for adding the collected data to the product image database when it is determined that the image is similar.

When the collected data is an image, the image analysis module extracts an object from the collected image, and determines whether the extracted object is similar to a product object of a product image stored in the product image database. The update module may link the collected image determined to be similar to the product object and add the same to the product image database.

If the collected data is a video, the image analysis module extracts an object from each frame of the collected video, determines whether the extracted object is similar to a commodity object of a commodity image stored in the commodity image database, and similarly. The frame image of the collected video determined to be determined may be extracted, and the database update module may add the extracted frame image to the product image database by linking with the product object.

If it is not determined that the image is similar, the image analysis module may delete the collected data or separately manage it as dissimilar data.

According to the present invention, the search server collects and analyzes data such as a video or an image from web servers in advance, and adds data similar to the product image stored in the product image database to improve the accuracy of image search and improve the user's search convenience. can do.

1 is a diagram illustrating an image search system according to an exemplary embodiment of the present invention.
2 is a block diagram showing the configuration of a search server according to an embodiment of the present invention.
3 is a flowchart illustrating an image retrieval method according to an embodiment of the present invention.
4 is a flowchart illustrating an image retrieval method according to an embodiment of the present invention.
5 is a flowchart illustrating an image retrieval method according to another embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to specific embodiments of the present invention shown in the accompanying drawings in order to clarify the features and advantages of the present invention.

However, in the following description and the accompanying drawings, detailed descriptions of well-known functions or configurations that may obscure the subject matter of the present invention will be omitted. In addition, it should be noted that like elements are denoted by the same reference numerals as much as possible throughout the drawings.

The terms or words used in the following description and drawings should not be construed as being limited to the common or dictionary meanings, and the inventors may appropriately define the concept of terms for describing their own invention in the best way. It should be interpreted as meanings and concepts corresponding to the technical idea of the present invention based on the principle that the present invention. Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical idea of the present invention, various modifications that can be substituted for them at the time of the present application It should be understood that there may be equivalents and variations.

In addition, terms including ordinal numbers, such as first and second, are used to describe various components, and are used only to distinguish one component from another component, and to limit the components. Not used. For example, without departing from the scope of the present invention, the second component may be referred to as the first component, and similarly, the first component may also be referred to as the second component.

In addition, when a component is referred to as being "connected" or "connected" to another component, it means that it may be connected or connected logically or physically. In other words, although a component may be directly connected or connected to other components, it should be understood that other components may exist in the middle and may be connected or connected indirectly.

In addition, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, the terms "comprises" or "having" described herein are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or the same. It is to be understood that the present invention does not exclude in advance the possibility of the presence or the addition of other features, numbers, steps, operations, components, parts, or a combination thereof.

In addition, the terms “… unit”, “… unit”, “module”, etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. have.

In addition, "a" or "an", "one", "the" and the like shall also be indicated otherwise in the context of the present invention (particularly in the context of the following claims). Unless otherwise expressly contradicted by context, it may be used in the sense including both singular and plural.

In addition, embodiments within the scope of the present invention include computer readable media having or conveying computer executable instructions or data structures stored on computer readable media. Such computer readable media can be any available media that can be accessed by a general purpose or special purpose computer system. By way of example, such computer readable media may be in the form of RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or computer executable instructions, computer readable instructions or data structures. And may include, but are not limited to, any other medium that can be used to store or deliver certain program code means, and can be accessed by a general purpose or special purpose computer system. .

In the following description and claims, “network” is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules. When information is transmitted or provided to a computer system via a network or other (wired, wireless, or a combination of wired or wireless) communication connections, this connection can be understood as a computer-readable medium. Computer-readable instructions include, for example, instructions and data that cause a general purpose or special purpose computer system to perform a particular function or group of functions. The computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, or even source code.

In addition, the present invention relates to personal computers, laptop computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers In a network computing environment having various types of computer system configurations, including (pager) and the like, it may be applied to providing advertisements to the computer systems. The invention may also be practiced in distributed system environments where both local and remote computer systems that are linked via wired networks, wireless data links, or combinations of wired and wireless data links perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

First, an image retrieval system using a query image according to an embodiment of the present invention will be described. 1 is a diagram illustrating an image search system according to an exemplary embodiment of the present invention. Referring to FIG. 1, an image search system includes a search server 100 and a user device 200.

The search server 100 and the user device 200 may exchange data with each other through a communication network NW. For example, the communication network (NW) may include a wireless communication network such as a wireless local area network (WLAN), Wi-Fi, Wibro, Wimax, High Speed Downlink Packet Access (HSDPA), Depending on how the system is implemented, it may also include wired communication networks such as Ethernet, xDSL (ADSL, VDSL), Hybrid Fiber Coaxial Cable (HFC), Fiber to The Curb (FTTC), and Fiber To The Home (FTTH).

In addition, the communication network NW of the present invention may include, for example, a mobile communication network including a plurality of access networks (not shown) and a core network (not shown) connecting them. Here, the access network is a network that performs wireless communication by directly connecting to a terminal, for example, a plurality of base stations such as a base station (BS), a base transceiver station (BTS), a NodeB, an eNodeB, and a base station controller (BSC). It may be implemented as a base station controller such as a Radio Network Controller (RNC). In addition, as described above, the digital signal processing unit and the wireless signal processing unit, which are integrally implemented in the base station, are divided into digital units (hereinafter referred to as DUs and radio units (hereinafter referred to as RUs)). In addition, a plurality of RUs (not shown) may be installed in each of the areas, and a plurality of RUs may be connected to a centralized DU, and a core network (not shown) constituting a mobile network together with an access network may be connected to an external network. It serves to connect networks, for example other communication networks, such as the Internet.

As described above, the core network is a network system that performs main functions for mobile communication services such as mobility control and switching between access networks, and performs circuit switching or packet switching, and performs mobile network. Manage and control the flow of packets within In addition, the core network may manage inter-frequency mobility and may play a role for interworking with traffic in the access network and the core network and other networks such as the Internet network. The core network may further include a serving gateway (SGW), a PDN gateway (PGW), a mobile switching center (MSC), a home location register (HLR), a mobile mobility entity (MME), and a home subscriber server (HSS). It may be.

In addition, the communication network (NW) according to the present invention may include an Internet network. The Internet network refers to a common public communication network, that is, a public network, through which information is exchanged according to the TCP / IP protocol. The search server 100 and the user device 200 interwork with each other through the communication network NW to form an image search system according to the present invention.

Here, the search server 100 is an entity existing on the network, and serves as a web server, a database server, and an application server. According to an exemplary embodiment, the search server 100 may search for and provide a web server that provides a shopping mall web site, a database server that stores an image of an item sold on the web site, and an image that is the same as or similar to a query image. Can act as an application server. In the following embodiment, it is assumed that the search server 100 is a server that provides an Internet shopping mall service that sells various products. Accordingly, the search server 100 may search for images in the plurality of images stored in the database. Such an image sold in an internet shopping mall may be a product image. Meanwhile, the search server 100 may search for images from a plurality of web pages by accessing another server, that is, a web server, through a network.

In addition, the user device 200 will be described as a representative example of the mobile communication terminal, but the terminal is not limited to the mobile communication terminal, but a variety of information communication devices, multimedia terminals, wired terminals, fixed terminals and IP (Internet Protocol) terminals, etc. Applicable to the terminal. For example, the terminal may be a mobile terminal having various mobile communication specifications such as a mobile phone, a portable multimedia player (PMP), a mobile internet device (MID), a smart phone, a tablet PC, a tablet PC, and an information communication device. Can be advantageously used. As another example, the terminal may be a notebook, a personal computer, or the like.

As described above, the search server 100 according to an embodiment of the present invention may be a server for providing an Internet shopping mall service. Accordingly, the search server 100 may store a plurality of product images in the database. The plurality of merchandise images have a quality standard of a predetermined level or more.

Meanwhile, the user device 200 may request an image search by transmitting a query image to the search server 100. For example, a user of the user device 200 reads the latest movie A on the web, captures the scene, and inputs the captured image as a query image to the search server 100 to search for the beverage C that famous actor B is drinking. can do. However, in this case, the search results for the famous actor B may be mainly provided rather than the search results for the beverage C.

As another example, a user of the user device 200 views a popular video D on the web, captures the scene, and inputs the captured image as a query image to the search server 100 to search for the sneaker F worn by the popular athlete E. can do. However, in this case, the search results for the popular athlete E may be mainly provided rather than the search results for the sneakers F.

As such, the user may view the latest movies, popular popular videos, commercial videos, trendy photos, etc. on the web, and capture specific scenes of interest and use them as query images. However, if the query image contains many objects, the user may not get the exact search results he wants.

This problem may occur even when the search server 100 is a server that provides an Internet shopping mall service. Although there is a product that the user wants to search among the products sold by the Internet shopping mall, the search server 100 may not provide an accurate search result quickly.

Accordingly, the search server 100 of the present invention collects materials such as a video or an image from web servers in advance, selects materials including specific objects, and adds them to a database to improve the accuracy of image search and to improve user's search convenience. It can be improved.

Hereinafter, the configuration of the search server 100 according to an embodiment of the present invention will be described. 2 is a block diagram showing the configuration of a search server according to an embodiment of the present invention. Referring to FIG. 2, the search server 100 may include a communication module 110, a storage module 120, and a control module 130.

The communication module 110 is for communication with the user device 200 and the web server 300. When the user device 200 accesses the search server 100, the communication module 110 performs communication for exchanging data including necessary information with the user device 200. The communication module 110 extracts data from the packet received from the user device 200 and transfers the extracted data to the control module 130. For example, the extracted data may be a query image. In addition, when the communication module 110 receives a search result including data for transmission from the control module 130 to the user device 200, the communication module 110 configures and transmits the received data into a packet. For example, the received data may be a plurality of search images that are image search results.

In addition, the communication module 110 communicates with the web servers 300 periodically or continuously. The communication module 110 transmits data downloaded from the web servers 300 to the control module 130. The downloaded material may be a video such as a movie, a video clip, an advertisement, or an image such as a photo.

The storage module 120 stores a program and data necessary for the operation of the search server 100 and may be divided into a program area and a data area. The program area may store a program for controlling the overall operation of the search server 100, an operating system (OS) for booting the search server 100, an application for image processing, an application for image search, and the like. The data area is an area in which data generated according to the operation of the search server 100 and data necessary for the operation of the search server 100 are stored. Various data stored in the storage module 120 may be deleted, changed, or added according to a user's manipulation.

The storage module 120 includes a data database 121 and a product image database 122. The data database 121 stores data collected from the web servers 300. In this case, the materials may be stored separately in a video and an image. In addition, the data may be selectively deleted according to a similarity analysis result to be described later, or may be classified and stored as similar or dissimilar data.

The product image database 122 may include category information and product images of products sold in an internet shopping mall. Such a product image may be an object of image retrieval and has a quality standard of a predetermined level or more.

The control module 130 may control an overall operation of the search server 100 and a signal flow between internal blocks of the search server 100 and may perform a data processing function for processing data. The control module 130 is preferably a central processing unit (CPU).

The control module 130 includes a data collection module 131, an image analysis module 132, a database update module 133, and an image search module 133.

The data collection module 131 periodically or continuously collects data including video and images from the web servers 300 through the communication module 110. The data collection module 131 stores the collected data in the data database 121.

The image analysis module 132 analyzes the data stored in the data database 121 and determines whether the image is similar to the product image stored in the product image database 122.

Specifically, the image analysis module 132 first collects whether the data collected from the web server 300 and stored in the data database 121 is an image or a video. If the collected data is an image, the image analysis module 132 extracts an object from the collected image. For example, if the popular athlete E has collected an image wearing the sneaker F, the image analysis module 132 extracts objects E and F from the image. The image analysis module 132 then determines whether the extracted object is similar to the product object of the product image stored in the product image database 122. If the image of the sneaker F is stored in the commodity image database 122, it may be determined that the object F is similar among the extracted objects E and F. FIG.

If there is an object determined to be similar among the extracted objects, the image analysis module 132 determines that the image is a similar image. Then, the database update module 133 links the collected image determined as a similar image with the product object to add to the product image database 122.

If there is no object determined to be similar among the extracted objects, the image analysis module 132 may determine the corresponding image as a dissimilar image and delete or separately manage it as a dissimilar image.

On the other hand, if the collected data is a video, the image analysis module 132 analyzes the collected video frame by frame. That is, the image analysis module 132 extracts an object into each frame of the collected video. For example, if a famous actor B is drinking a drink C in a specific frame of the latest movie A collected, the image analysis module 132 extracts objects B and C from the frame. The image analysis module 132 then determines whether the extracted object is similar to the product object of the product image stored in the product image database 122. If an image of the beverage C is stored in the product image database 122, it may be determined that the object C is similar among the extracted objects B and C. FIG.

If there is an object determined to be similar among the extracted objects, the image analysis module 132 determines that the video frame is a similar image and extracts the frame image from the video. Then, the database update module 133 determines the similar image and adds the extracted frame image to the product image database 122 by linking with the product object.

In this manner, frame images similar to the plurality of product objects may be independently extracted for each frame of the collected video and added to the product image database 122. If all frames of the collected video do not include an object determined to be similar, the image analysis module 132 may determine the video as similar data and delete or separately manage the video as similar data.

The image retrieval module 133 performs image retrieval from the commodity image database 122 using the query image when the query image is input from the user device 200. Then, the image search module 133 transmits a search result including the plurality of search images to the user device 200 through the communication module 110.

Hereinafter, an image search method according to an embodiment of the present invention will be described. 3 is a flowchart illustrating an image retrieval method according to an embodiment of the present invention. 4 is a flowchart illustrating an image retrieval method according to an embodiment of the present invention. 5 is a flowchart illustrating an image retrieval method according to another embodiment of the present invention.

First, referring to FIG. 3 together with FIG. 2 described above, the data collection module 131 of the search server 100 periodically retrieves data including video and images from the web servers 300 through the communication module 110. Or continuously (step S310). And the data collection module 131 stores the collected data in the data database 121 (step S320).

Subsequently, the image analysis module 132 of the search server 100 determines whether the data collected from the web server 300 and stored in the data database 121 is an image or a video (step S330). Subsequent processes when the material is divided into images at this stage are shown in FIG. 4, and subsequent processes when the material is divided into moving images are shown in FIG. 5. This will be described later.

Subsequently, the image analysis module 132 analyzes the data stored in the data database 121 to determine whether the image is similar to the product image stored in the product image database 122 (step S340).

If it is determined in step S340 that is similar, the database update module 133 of the search server 100 adds the collected image determined as a similar image to the product image database 122 by linking with the product object (step S350).

If it is determined in step S340 that the dissimilarity, the image analysis module 132 may determine the image as similar data and delete or separately manage it as dissimilar data (step S360).

Next, a case in which the collected data is an image will be described with reference to FIG. 4.

If it is determined that the collected data is an image (step S410), the image analysis module 132 extracts an object from the collected image (step S420). For example, if the popular athlete E has collected an image wearing the sneaker F, the image analysis module 132 extracts objects E and F from the image.

Then, the image analysis module 132 determines whether the extracted object is similar to the product object of the product image stored in the product image database 122 (step S430). If the image of the sneaker F is stored in the commodity image database 122, it may be determined that the object F is similar among the extracted objects E and F. FIG.

If there is an object determined to be similar among the extracted objects, the image analysis module 132 determines that the image is a similar image. Then, the database update module 133 links the collected image determined as a similar image with the product object and adds it to the product image database 122 (step S440).

If there is no object determined to be similar among the extracted objects, the image analysis module 132 may determine the corresponding image as dissimilar data and delete or separately manage it as dissimilar data (step S450).

Next, a case in which the collected data is a moving picture will be described with reference to FIG. 5.

If it is determined that the collected data is a video (step S510), the image analysis module 132 analyzes the collected video on a frame basis. That is, the image analysis module 132 extracts an object into each frame of the collected video (step S520). For example, if a famous actor B is drinking a drink C in a specific frame of the latest movie A collected, the image analysis module 132 extracts objects B and C from the frame.

Subsequently, the image analysis module 132 determines whether the extracted object is similar to the product object of the product image stored in the product image database 122 (step S530). If an image of the beverage C is stored in the product image database 122, it may be determined that the object C is similar among the extracted objects B and C. FIG.

If there is an object determined to be similar among the extracted objects, the image analysis module 132 determines that the video frame is a similar image and extracts the frame image from the video (step S540).

Then, the database update module 133 determines the similar image and adds the extracted frame image to the product image database 122 by linking with the product object (step S550).

In this manner, frame images similar to the plurality of product objects may be independently extracted for each frame of the collected video and added to the product image database 122. If all the frames of the collected video do not include the object determined to be similar, the image analysis module 132 may determine the video as dissimilar data and delete or separately manage it as dissimilar data (step S560). .

As described above, the specification includes the details of a number of specific implementations, but these should not be understood as being limited to the scope of any invention or claimable, but rather may be specific to a particular embodiment of a particular invention. It should be understood as a description of the features present. Certain features that are described in this specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, while the features operate in a specific combination and may be initially depicted as so claimed, one or more features from the claimed combination may in some cases be excluded from the combination, wherein the claimed combination is a subcombination Or a combination of subcombinations.

Likewise, although the operations are depicted in the drawings in a specific order, it should not be understood that such operations must be performed in the specific order or sequential order shown in order to obtain desirable results or that all illustrated operations must be performed. In certain cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that it can.

Specific embodiments of the subject matter described in this specification have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order but still achieve desirable results. As an example, the process depicted in the accompanying drawings does not necessarily require that particular illustrated or sequential order to obtain the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

The foregoing description presents the best mode of the invention, and provides examples to illustrate the invention and to enable those skilled in the art to make and use the invention. The specification thus produced is not intended to limit the invention to the specific terms presented. Thus, while the present invention has been described in detail with reference to the examples described above, those skilled in the art can make modifications, changes and variations to the examples without departing from the scope of the invention.

Therefore, the scope of the present invention should be determined by the claims rather than by the described embodiments.

100: search server 110: communication module
120: storage module 121: data DB
122: product image DB 130: control module
131: data collection module 132: image analysis module
133: DB Update Module 134: Image Search Module
200: user device 300: web server

Claims (8)

Collecting, by the data collection module, data including a video and an image from a web server;
Analyzing, by the image analysis module, the collected data to determine whether the image is similar to a product image stored in a product image database; And
If it is determined that the image is similar, the database updating module adds the collected data to the product image database;
Image search method comprising a. The method of claim 1,
If the collected material is an image,
The determining may include extracting an object from the collected image by the image analysis module, and determining whether the extracted object is similar to a product object of a product image stored in the product image database.
The adding may include linking the collected image determined to be similar to the database update module with the product object to add to the product image database. The method of claim 1,
If the collected data is a video,
The determining may include: extracting an object from each frame of the collected video by the image analysis module, determining whether the extracted object is similar to a product object of a product image stored in the product image database; Extracting a corresponding frame image of the collected video determined to be similar;
The adding may include linking the extracted frame image with the commodity object and adding the extracted frame image to the commodity image database. The method of claim 1,
If it is not determined that the image is similar, the image analysis module deleting the collected data or separately managing the dissimilar data;
Image retrieval method further comprising. A communication module for communicating with a plurality of web servers;
A product image database for storing a plurality of product images;
A data collection module for collecting data including video and images from the web servers through the communication module;
An image analysis module for analyzing the collected data and determining whether the image is similar to a product image stored in the product image database; And
A database update module for adding the collected data to the product image database if it is determined that the image is similar;
Image retrieval device comprising a. The method of claim 5,
If the collected material is an image,
The image analysis module extracts an object from the collected image, and determines whether the extracted object is similar to a product object of a product image stored in the product image database.
And the database update module adds the collected image determined to be similar to the product image database by linking with the product object. The method of claim 5,
If the collected data is a video,
The image analysis module extracts an object from each frame of the collected video, determines whether the extracted object is similar to a commodity object of a commodity image stored in the commodity image database, and corresponds to the corresponding collected video determined to be similar. Extract the frame image,
And the database update module adds the extracted frame image to the product image database by linking with the product object. The method of claim 5,
If it is not determined that the similar image, the image analysis module, characterized in that for deleting the collected data or separately managed as dissimilar data.

Images