KR20200141387A - System, method and program for searching image data by using deep-learning algorithm - Google Patents

Abstract

The present invention relates to an image data search method using a deep learning algorithm, to a device thereof, and to a program. According to one embodiment of the present invention, the image data search method comprises the steps of: calculating individual appearance characteristics for a plurality of appearance classification criteria by inputting input image data into an appearance characteristic recognition model by a server (S200); generating, by the server, appearance description data by combining a plurality of individual appearance characteristics with respect to the input image data (S400); and extracting the image data corresponding to a combination of the appearance classification criteria matched with an abstract characteristic corresponding to a search keyword in a matching algorithm by receiving of a search keyword from a specific user by the server (S600).

Description

Image data search method, device and program using deep learning algorithm {SYSTEM, METHOD AND PROGRAM FOR SEARCHING IMAGE DATA BY USING DEEP-LEARNING ALGORITHM}

The present invention relates to an image data search method, apparatus, and program using a deep learning algorithm.

With the recent development of the Internet, social media network services are rapidly growing. As a result, as the amount of multimedia is explosively increased, an effective image retrieval system is required, and image annotations are becoming increasingly important due to the necessity of efficient image retrieval according to explosively increasing web images.

Most of the image retrieval studies have been mainly conducted on a Content-based Image Retrieval (CBIR) method that analyzes the contents of an image. Content-based image search analyzes the content of an image using visual features such as color, texture, and shape. This method works well when the number of defined tags is small, but performance deteriorates as the data set becomes larger and the types of tags diversify.

Text-based Image Retrieval (TBIR) is a method of searching for images corresponding to text by querying text. In this method, the visual content of the image is represented by a manually tagged text descriptor, and is used to perform image search in a database management system. That is, in the existing image or video search method, a search is performed based on information directly tagged by a user. There is a problem that the search result is inaccurate when the user incorrectly tags the keyword in the video. In addition, since there may be differences in keywords defined for each user, there is a problem that the results provided according to the keywords selected by the user searching for images are different.

An object of the present invention is to provide an image data search method, apparatus, and program using a deep learning algorithm that accurately extracts image data desired by a user based on an abstract term representing the appearance of a specific object.

In addition, the present invention provides an image data retrieval method, apparatus, and program using a deep learning algorithm that can be transformed with a minimum amount of additional training data in a situation where a transformation is required in an external characteristic recognition model such as a change in the definition of an abstract term. I want to provide.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

An image data retrieval method according to an embodiment of the present invention includes the steps of: inputting the input image data into an appearance characteristic recognition model by a computer, and calculating individual appearance characteristics for a plurality of appearance classification criteria; Generating, by a computer, appearance description data by combining a plurality of individual appearance characteristics with respect to the input image data; And extracting image data corresponding to a combination of appearance classification criteria matched with an abstract characteristic corresponding to the search keyword from a matching algorithm, as the computer receives a search keyword from a specific user. Including, the appearance classification criteria Is a specific classification standard for describing the appearance of a specific object, and includes a plurality of individual external characteristics expressing various external characteristics within the same classification standard of the object.

In another embodiment, the appearance characteristic recognition model includes a plurality of individual characteristic recognition modules for determining different appearance classification criteria, and the individual characteristic recognition module is an individual appearance characteristic included in a specific appearance classification standard of image data. In the step of calculating the individual appearance characteristics, the input image data is input to each individual characteristic recognition module in the appearance characteristic recognition model to calculate a plurality of individual appearance characteristics for the input image data. To do.

In addition, in another embodiment, the individual characteristic recognition module is trained through a deep learning learning model by matching individual appearance characteristics of a specific appearance classification criterion with respect to a plurality of training image data.

In addition, in another embodiment, the external feature recognition model is characterized in that it includes a combination of individual feature recognition modules different for each object type.

In addition, in another embodiment, when there is an unlearned appearance classification criterion for a specific object in which the individual characteristic recognition module is not built, the step of generating the appearance description data is performed by combining the input individual appearance characteristics and the calculated individual appearance characteristics. Create appearance description data, wherein the input individual appearance characteristics are obtained for the unlearned appearance classification criteria from an image provider client or expert client who provided the input image data, and the calculated individual appearance characteristics are the individual characteristics recognition module. It is calculated by inputting the input image data to.

In addition, in another embodiment, when a new appearance classification standard for a specific object is added, the computer obtains individual appearance characteristics of the new appearance classification standard for the training image data, and constructs a new learning data set; And training, by the computer, a new individual feature recognition module based on the new learning data set, and adding it to the external feature recognition model.

In addition, in another embodiment, the step of adding detailed information for the new appearance classification criteria by inputting the input image data, in which the appearance description data has been obtained by the individual characteristic recognition module already built, into a new individual characteristic recognition module. ; And updating the matching algorithm by matching, by a computer, individual external characteristics of the new external appearance classification criteria with each abstract characteristic.

In another embodiment, the method further includes: receiving, by a computer from an expert client, setting data matching the combination of the abstract characteristic and the appearance classification criteria, and setting the matching algorithm.

In another embodiment, the step of generating the appearance description data may include extracting code values corresponding to a plurality of individual appearance characteristics of the input image data; And generating external description data in the form of a code string by combining the plurality of code values.

In another embodiment, a reference image data acquiring step, which is a reference for searching for an image in which a computer obtains reference image data from a user client, the reference image data having similar appearance characteristics among the same object; Inputting the reference image data into the appearance characteristic recognition model to calculate individual appearance characteristics for a plurality of appearance classification criteria; Generating, by a computer, appearance description data by combining a plurality of individual appearance characteristics with respect to the reference image data; And extracting, by a computer, image data including the same outline description data as the reference image data.

In another embodiment, as the computer receives a request to provide additional image data from a user client, sequentially providing image data having different appearance classification criteria; And when one or more image data is selected from the additional image data by the user, the computer setting a personalized abstract characteristic based on the appearance description data of the selected image data.

Further, as another embodiment, when there is an abstract characteristic corresponding to the outline description data of the selected image data, providing, by a computer, an abstract characteristic suitable for extracting the selected image data to a user client.

In another embodiment, when the image data is moving image data including a plurality of frames, the step of calculating the individual appearance characteristics is performed for each frame in the moving image data, and the appearance description The data generation step is characterized in that it is generated by sequentially listing a plurality of individual appearance characteristics for each frame, and the abstract characteristic is an expression representing a specific shape change or motion.

The image data search program according to another embodiment of the present invention is combined with hardware to execute the aforementioned image data search method, and is stored in a medium.

An image data search apparatus according to another embodiment of the present invention includes one or more computers, and performs the aforementioned image data search method.

According to the present invention as described above, it has various effects as follows.

First, when searching for an abstract characteristic used to express the appearance of each object (for example, an emotional term such as'vintage' in fashion clothing), image data matching the abstract characteristic will be automatically extracted and provided. I can.

Second, by learning each individual feature recognition module, when new input image data is acquired (for example, when new products are continuously added to the shopping mall page), the appearance of each input image data is described without additional learning. Data can be created to enable search based on abstract characteristics.

Third, it is possible to minimize the situation in which the external characteristic recognition model needs to be re-learned or additionally learned. For example, if the definition of the external characteristics of an abstract term is different, only the individual external characteristic combination (i.e., external description information) matched with the abstract term (i.e., abstract characteristic) is modified without the need to perform new learning, Search can be made according to the changed definition. That is, even if the definition of the abstract feature is different, it is not necessary to additionally learn the individual feature recognition module. In addition, for example, when a new appearance classification criterion is added for a specific object, only a new individual characteristic recognition module needs to be learned, and the existing individual characteristic recognition module can be used as it is.

1 is a block diagram of an image data search system according to an embodiment of the present invention.
2 and 4 to 11 are flow charts of a method for searching image data according to embodiments of the present invention.
3 is an exemplary diagram of a server including an external characteristic recognition model for each object according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments to be posted below, but may be implemented in a variety of different forms, and only these embodiments make the posting of the present invention complete, and common knowledge in the art It is provided to completely inform the scope of the invention to those who have, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements.

In the present specification, a'computer' includes all various devices capable of performing arithmetic processing and providing results to a user. For example, computers are not only desktop PCs and notebooks, but also smart phones, tablet PCs, cellular phones, PCS phones, and synchronous/asynchronous systems. A mobile terminal of the International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a personal digital assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to the server 10 that receives a request from a client and performs information processing.

In the present specification, "object" refers to an article of a specific classification or category for performing a search. For example, when a user wants to search for an image of a desired item in a shopping mall, when a user searches for clothes among item categories, the object may be clothes.

In the present specification,'image data' means a two-dimensional or three-dimensional static or dynamic image including a specific object. That is,'image data' may be static image data that is one frame, or dynamic image data (ie, moving image data) in which a plurality of frames are consecutive.

In the present specification,'learning image data' means image data used for training a learning model.

In this specification,'input image data' is image data that is input to a learning model to obtain a learning result.

In the present specification,'reference image data' means image data that is a reference of an object to be searched by a user. That is, the'reference image data' is a criterion for searching for images including those having similar appearance characteristics among the same objects. For example, when a user wants to search for something similar to an external shape of an object included in a specific image, the reference image data may be an image input by the user.

In the present specification, the'appearance classification standard' refers to a classification standard of an appearance expression necessary for describing the appearance of a specific object or for annotation. That is, the'appearance classification standard' is a specific classification standard for describing the appearance of a specific object, and includes a plurality of individual appearance characteristics expressing various external characteristics within the same classification standard of the object. For example, when the object is clothing , The appearance classification standard is a classification standard for the appearance of clothing, and may correspond to a pattern, color, fit, length, and the like. That is, when the appearance classification standard for a specific object increases, the external shape of a specific article belonging to the object can be described in detail.

In this specification,'individual appearance characteristics' refers to various characteristics included in a specific appearance classification standard. For example, if the appearance classification criterion is color, the individual appearance characteristics mean various individual colors.

In the present specification, the'image provider client 40' refers to a client of an image provider that uploads specific image data to the server 10. For example, when uploading an image of a product for sale to a shopping mall, the image provider client 40 is a client of a person who uploads the image to the shopping mall.

In this specification, the'user client 20' refers to a client of a user who searches for image data.

In this specification, the'expert client 30' gives individual appearance characteristics to the learning image data (i.e., labeling the learning image data) or gives the input image data individual appearance characteristics within the unlearned appearance classification criteria. It means the client of the expert who plays the role.

In the present specification, "abstract characteristic" refers to an abstract characteristic given to a specific object. For example, the'abstract characteristic' may be an emotional characteristic for a specific object (for example, in the case of clothing, an emotional or fashionable expression such as vintage). In addition, for example,'abstract characteristic' may mean a shape change or motion when the image data is a moving picture.

Hereinafter, a detailed description of an image data search apparatus, method, and program according to embodiments of the present invention will be described with reference to the drawings.

1 is a block diagram of an image data search system according to an embodiment of the present invention.

2 is a flowchart of an image data search method according to an embodiment of the present invention.

1 and 2, in the image data search method according to an embodiment of the present invention, the server 10 inputs the input image data into the appearance characteristic recognition model 100, and the plurality of appearance classification criteria Calculating individual appearance characteristics (S200); Generating, by the server 10, external appearance description data by combining a plurality of individual appearance characteristics of the input image data (S400); And extracting image data corresponding to a combination of appearance classification criteria matched with an abstract characteristic corresponding to the search keyword in the matching algorithm 200 as the server 10 receives a search keyword from a specific user (S600). Includes; Hereinafter, a detailed description of each step will be described.

The server 10 inputs the input image data into the appearance characteristic recognition model 100 to calculate individual appearance characteristics for a plurality of appearance classification criteria (S200). That is, the server 10 provides new input image data for which the external characteristic analysis has not been performed to the external characteristic recognition model 100 to calculate individual external characteristics for each external classification standard of a specific object.

In one embodiment, the appearance characteristic recognition model 100 includes a plurality of individual characteristic recognition modules 110 for determining different appearance classification criteria, as shown in FIG. 1. That is, the appearance characteristic recognition model 100 includes a plurality of individual characteristic recognition modules 110 specialized to recognize each appearance classification criterion. The more the external classification criteria of a specific object are, the more the server 10 includes a plurality of individual characteristic recognition modules 110 in the external characteristic recognition model 100. The individual characteristic recognition module 110 calculates individual external characteristics included in a specific external classification standard of image data.

In one embodiment, in the step of calculating the individual appearance characteristics (S200), by inputting input image data to each individual characteristic recognition module 110 in the appearance characteristic recognition model 100, a plurality of Calculate individual appearance characteristics. Through this, the server 10 acquires all the individual appearance characteristics of each appearance classification criterion for the input image data.

In addition, in an embodiment, the individual characteristic recognition module 110 is trained through a deep learning learning model by matching individual external characteristics of a specific external classification standard with respect to a plurality of training image data. That is, the individual characteristic recognition module 110 is constructed with a specific deep learning algorithm, and learning is performed by matching a specific one of a plurality of appearance classification criteria with image data for learning.

To this end, the server 10 may perform a process of training each individual characteristic recognition module 110 as follows.

First, the server 10 acquires a plurality of training image data for a specific object. For example, when the object is a specific type of clothing (eg, a shirt), the server 10 acquires images of several shirts. The server 10 may be selected by an expert from among image data previously stored, or may newly acquire an object image that is easy to learn.

The server 10 obtains a plurality of individual appearance characteristics for each appearance classification standard definition and each appearance classification standard. That is, the server 10 sets the initial number of individual characteristic recognition modules 110 according to the setting of a plurality of appearance classification criteria. In addition, the server 10 sets a type of feature for labeling the training image data for each appearance classification criterion as a plurality of individual appearance characteristics in each appearance classification criterion are set.

In an embodiment, the server 10 may receive a plurality of appearance classification criteria for analyzing the appearance of a specific object and a plurality of individual appearance characteristics within each appearance classification standard from the expert client 30 for analyzing the appearance of a specific object. . For example, when constructing the outer appearance characteristic recognition model 100 for clothing, the server 10 may receive an appearance classification standard and individual appearance characteristics included therein from a client of a designer who is a clothing expert.

Thereafter, the server 10 labels the training image data with a plurality of individual appearance characteristics of each appearance classification criterion. That is, the server 10 receives and matches at least one individual appearance characteristic for each of a plurality of appearance classification criteria for each training image data. For example, when 10 external classification criteria are set for a specific object, the server 10 receives one individual external characteristic for each of the 10 external classification criteria for each learning image data including the corresponding object, and A training data set that matches the image data and 10 individual type characteristics is formed.

Thereafter, the server 10 performs training by matching the training image data with individual appearance characteristics of a specific external classification standard labeled therefor. That is, when the server 10 trains the individual characteristic recognition module 110 for the A appearance classification criterion, deep learning by extracting only the training image data and the individual appearance characteristics of the A appearance classification criterion matched from the training dataset. Enter into the learning model. Through this, the server 10 constructs each individual characteristic recognition module 110 capable of recognizing individual external characteristics of each external classification standard.

In addition, in another embodiment, the external feature recognition model 100 includes a combination of individual feature recognition modules 110 different for each object type, as shown in FIG. 3. For example, fashion miscellaneous goods types (for example, shoes, wallets, bags) belonging to the same major category have different appearance classification criteria, so the server 10 creates a combination of individual characteristic recognition modules 110 for each object type. Thus, a specialized external feature recognition model for recognizing the external appearance of a specific object is created.

In addition, in another embodiment, each external characteristic recognition model 100 for a plurality of objects may share and use a specific individual characteristic recognition module 110. For example, when the individual characteristic recognition module 110 performs color recognition, the color recognition module can be used universally regardless of the object type, so that the server 10 provides a plurality of external characteristics distinguished for each object. A universal color recognition module can be used in the recognition model 100.

The server 10 generates appearance description data by combining or listing a plurality of individual appearance characteristics for the input image data (S400). If the external appearance classification criteria for a specific object are divided in detail, the external appearance description data specifically describes the external appearance of the object through individual external characteristics.

In another embodiment, the step of generating the appearance description data (S400) may include extracting a code value corresponding to a plurality of individual appearance characteristics of the input image data (S410); And generating external description data in the form of a code string by combining the plurality of code values (S411). That is, as the server 10 codes the individual appearance characteristics, the appearance description data can be generated as a code string, and through this, the processing of the appearance description data can be efficiently performed.

In addition, as another embodiment, when there is an unlearned external classification standard of a specific object for which the individual characteristic recognition module 110 is not constructed (for example, recognition through a deep learning learning model among external classification standards of the object) If there is something difficult to do or the individual characteristic recognition module 110 has not yet been constructed due to the creation of a new external classification standard), the server 10 is based on the unlearned external classification standard from the expert client or the image provider client 40. For each, the individual appearance characteristics of the input image data are input.

Specifically, in the appearance description data generation step (S400), the server 10 generates the appearance description data by combining the input individual appearance characteristics and the calculated individual appearance characteristics. The input individual appearance characteristics are obtained for the unlearned appearance classification criteria from an image provider client 40 or an expert client that provided the input image data, and the calculated individual appearance characteristics are obtained from the individual characteristic recognition module 110. It is calculated by inputting the input image data.

As the server 10 receives a search keyword from a specific user, the matching algorithm 200 extracts image data corresponding to a combination of appearance classification criteria matched with an abstract characteristic corresponding to the search keyword (S600). When a user wants to search for desired image data based on a search keyword that is one of the abstract characteristics of a specific object or a search keyword that is similar to the abstract characteristic, the server 10 uses the matching algorithm 200 to search for the desired image data. A combination of appearance classification criteria matching the corresponding abstract characteristics is extracted, and image data having a corresponding combination of appearance classification criteria in the appearance description data is extracted.

In one embodiment, the abstract characteristic may be matched with a plurality of individual appearance characteristics for a specific appearance classification criterion. In addition, when a specific appearance classification standard is not considered in defining a specific abstract characteristic, the server 10 may not match the specific external classification standard with the corresponding abstract characteristic. For example, when defining the abstract characteristic X, the appearance classification criterion 1 does not need to be considered (that is, when an object to which all the individual appearance characteristics of the appearance classification criterion 1 is applied can be included in the abstract characteristic X), the server ( 10) may not match the appearance classification criterion 1 with the abstract characteristic X. In addition, the server 10 may match a plurality of individual appearance characteristics of the appearance classification criterion 2 with the abstract characteristic X.

In addition, as another embodiment, as shown in FIG. 5, when a new appearance classification standard for a specific object is added, the server 10 acquires individual appearance characteristics of the new appearance classification standard for the learning image data, Building a learning data set (S700); And training, by the server 10, a new individual characteristic recognition module 110 based on the new learning data set, and adding it to the external characteristic recognition model (S710). That is, when a new external classification standard for a specific object is added (for example, a new standard for dividing the external characteristics of clothing is added), the server 10 does not change the existing individual characteristic recognition module 110. It is possible to change the external characteristic recognition model 100 according to the situation in which the new external classification standard is added by additionally constructing only the individual characteristic recognition module 110 for the new external classification standard.

First, the server 10 acquires individual appearance characteristics of the new appearance classification criteria for the training image data, and constructs a new training data set (S700). In one embodiment, when constructing a new individual characteristic recognition module 110 by using the same image data used to train another individual characteristic recognition module 110 in the past, the server 10 is an expert client 30 ), the individual appearance characteristics of the new appearance classification standard are input for each learning image data. In addition, in another embodiment, the server 10 acquires new image data for training the individual characteristic recognition module 110 for the new appearance classification criteria, and inputs individual appearance characteristics of the new appearance classification criteria, respectively. And construct a new training data set.

Thereafter, the server 10 trains the new individual feature recognition module 110 based on the new learning data set, and adds it to the external feature recognition model (S710). Through this, the server 10 adds a new individual characteristic recognition module 110 together with a plurality of existing individual characteristic recognition modules 110 to the existing plurality of external characteristic recognition models.

In addition, in another embodiment, as shown in FIG. 6, the input image data obtained by the external description data obtained by the individual characteristic recognition module 110 in which the server 10 has already been constructed is a new individual characteristic recognition module 110. It further includes a step (S720) of inputting and adding individual appearance characteristics for the new appearance classification criteria. That is, the server 10 performs a process of updating the appearance description data of the previously acquired input image data to reflect the new appearance classification criteria. To this end, the server 10 inserts all the input image data into the new individual characteristic recognition module 110 to calculate the individual appearance characteristics.

In another embodiment, as shown in FIG. 7, the server 10 updating the matching algorithm 200 by matching the individual appearance characteristics of the new appearance classification criteria with each abstract characteristic (S730); It includes more. That is, when the user searches for image data based on a keyword corresponding to an abstract characteristic, the server 10 reflects the new appearance classification criteria and provides the optimal search result. For characteristics, the individual external characteristics of the new external classification standard are linked.

In another embodiment, as shown in FIG. 8, the server 10 receives setting data matching the combination of the abstract characteristic and the appearance classification criteria from an expert client, and sets the matching algorithm 200 ( S800); further includes. The definition of abstract characteristics may be changed or different due to factors such as regional differences, changes in the times, and establishment of new definitions. For example, when the object is fashion clothing or fashion accessories, abstract characteristics representing specific fashion trends or emotional characteristics may change according to the change of the times, and may be defined differently according to regions around the world (for example, ' The abstract characteristic (ie, emotional characteristic) of'vintage' can be defined as having a different appearance in the past and the present.) Therefore, the server 10 is a matching relationship between the combination of abstract characteristics in the matching algorithm 200 and individual appearance characteristics. You can add or change settings.

In one embodiment, when the definition of a specific abstract characteristic is changed, the server 10 receives a combination of appearance classification criteria for the current abstract characteristic from the expert client 30. In this case, the server 10 may set the combination of the abstract characteristic before the change and the appearance classification criterion as the definition of the corresponding abstract characteristic at a specific point in the past. Through this, the server 10 may accumulate definition or description information of specific abstract characteristics according to changes in the times.

In addition, in another embodiment, since the same abstract characteristic for each region must be defined as a different appearance, the server 10 may receive and store a combination of external classification criteria for each region from the expert client 30.

In addition, as another embodiment, as shown in FIG. 9, the server 10 obtaining reference image data from the user client 20 (S900; reference image data acquisition step); Inputting the reference image data into the appearance characteristic recognition model 100 to calculate individual appearance characteristics for a plurality of appearance classification criteria (S910); Generating, by the server 10, appearance description data by combining a plurality of individual appearance characteristics with respect to the reference image data (S920); And extracting, by the server 10, image data including appearance description data identical or similar to the reference image data (S930). That is, when a user does not search based on a keyword corresponding to an abstract characteristic, but performs a search based on a specific object image (ie, reference image data) that the user has, the server 10 Appearance description data is generated, and image data including the same or similar appearance description data is extracted and provided to the user client 20.

First, the server 10 acquires the reference image data from the user client 20 (S900). That is, the server 10 receives the reference image data stored in the user client 20 or searched online by the user.

Thereafter, the server 10 inputs the reference image data into the appearance characteristic recognition model 100 to calculate individual appearance characteristics included in each appearance classification criterion (S910). That is, the server 10 acquires a plurality of individual external characteristics for describing the external characteristics of the reference image data as text information through each individual characteristic recognition module 110.

Thereafter, the server 10 generates appearance description data by combining a plurality of individual appearance characteristics with respect to the reference image data (S920).

After that, the server 10 extracts image data including the same outline description data as the reference image data. In one embodiment, in the case of searching for image data including outline description data identical to the reference image data, the server 10 searches for and provides input image data having the same outline description data as the reference image data.

In addition, in another embodiment, in the case of searching for image data to a range similar to the reference image data, the server 10 includes from a low importance to a similar range among a plurality of appearance classification criteria included in the appearance description data of the reference image data. It expands and extracts image data including one or more extended outline description data. To this end, the server 10 may include an importance ranking for a plurality of appearance classification criteria of a specific object (for example, the higher the importance ranking, the higher the priority, the higher the search range is maintained at a fixed value), The degree of similarity between individual appearance characteristics within a specific appearance classification criterion may be included.

In another embodiment, as shown in FIG. 10, as the server 10 receives a request for providing additional image data from the user client 20, at least one external classification criterion is sequentially provided. Step S1000; And when one or more image data is selected from the additional image data by the user, the server 10 setting a personalized abstract characteristic based on the appearance description data of the selected image data (S1010). That is, when performing a search based on a search keyword, the server 10 expands the search range and provides additional image data while changing at least one appearance classification criterion from the description information of the abstract characteristic corresponding to the search keyword to another individual appearance characteristic. Provided to the user client 20. After that, the server 10 receives one or more desired image images from the extended search range from the user. Thereafter, the server 10 personalizes a search keyword or abstract characteristic input by the user based on the selected video image. For example, since the external definition of the general abstract characteristic and the external definition of the abstract characteristic that the user is thinking may be different, the server 10 is based on the external description data of the video image selected by the user in the expanded search result. Set the description information or appearance definition of the abstract characteristic that the user thinks (that is, the description information of the personalized abstract characteristic). Through this, if the user performs a search with the same search keyword or abstract characteristic in the future, the server 10 does not search based on the description information of the general abstract characteristic, but performs a search based on the description information of the personalized abstract characteristic. By doing so, the user can first provide the desired image.

In addition, in another embodiment, as shown in FIG. 11, when there is an abstract characteristic corresponding to the outline description data of the selected image data, the server 10 provides the user client 20 to the user client 20 for extracting the selected image data. It further includes; providing an abstract characteristic (S1020). That is, the server 10 notifies that the external appearance definition known to the user for a specific abstract characteristic is different from the commonly used appearance definition, and provides an abstract characteristic (or search keyword) that matches the external appearance definition that the actual user thinks. It is extracted and provided. Through this, it is possible for the user to recognize a search keyword for obtaining a desired search result when re-searching later.

In addition, in another embodiment, when the image data is moving image data including a plurality of frames, the abstract characteristic may be an expression indicating a specific shape change or motion. That is, the abstract characteristic may be a textual expression representing a specific motion or shape change.

To this end, the server 10 generates appearance description data in which combinations of individual appearance characteristics (that is, individual appearance characteristics belonging to each appearance classification criterion) for a plurality of frames of input image data, which are moving images, are arranged in time series. Specifically, the individual appearance characteristic calculation step (S200) is performed for each frame in the moving picture data, and the appearance description data generation step (S400) sequentially lists a plurality of individual appearance characteristics for each frame. Generated by

In addition, in another embodiment, the server 10 is a matching algorithm 200 in which each abstract characteristic (for example, an expression representing a shape change or motion) and time series data of individual appearance characteristics within each appearance classification criterion are matched. Includes. Through this, in the image data search step (S600), the server 10 searches for and provides video data corresponding to an abstract characteristic (ie, a specific motion or shape change) desired by the user.

An image data search apparatus according to another embodiment of the present invention includes one or more computers, and performs the aforementioned image data search method.

The image data retrieval method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

The above-described program includes C, C++, JAVA, machine language, etc. that can be read by the computer's processor (CPU) through the computer's device interface in order for the computer to read the program and execute the methods implemented as programs It may include a code (Code) coded in the computer language of. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server 10 at a remote location in order to execute the functions, the code may use the communication module of the computer It may further include a communication related code for how to communicate with the server 10 or the like, and what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers 10 to which the computer can access, or on various recording media on the computer of the user. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

Claims (15)

Calculating individual appearance characteristics for a plurality of appearance classification criteria by inputting the input image data into an appearance characteristic recognition model by the server;
Generating, by a server, appearance description data by combining a plurality of individual appearance characteristics with respect to the input image data; And
In response to the server receiving a search keyword from a specific user, extracting image data corresponding to a combination of appearance classification criteria matched with an abstract characteristic corresponding to the search keyword from a matching algorithm; Including,
The appearance classification standard is a specific classification standard for describing the appearance of a specific object, and includes a plurality of individual appearance characteristics expressing various external appearance characteristics within the same classification standard of the object. The method of claim 1,
The appearance characteristic recognition model includes a plurality of individual characteristic recognition modules for determining different appearance classification criteria,
The individual characteristic recognition module is to calculate individual appearance characteristics included in a specific appearance classification standard of image data,
The step of calculating the individual appearance characteristics,
And calculating a plurality of individual appearance characteristics for the input image data by inputting the input image data into each individual characteristic recognition module in the appearance characteristic recognition model. The method of claim 2,
The individual characteristic recognition module,
An image data search method that is trained through a deep learning learning model by matching individual appearance characteristics of a specific appearance classification standard with respect to a plurality of training image data. The method of claim 2,
The external feature recognition model, characterized in that it comprises a combination of individual feature recognition modules that are different for each object type. The method of claim 2,
If there is an unlearned appearance classification criterion for a specific object for which the individual characteristic recognition module is not established,
The step of generating the appearance description data,
Appearance description data is generated by combining the input individual appearance characteristics and the calculated individual appearance characteristics, and the input individual appearance characteristics are obtained for the unlearned appearance classification criteria from the image provider client or expert client that provided the input image data. , The calculated individual appearance characteristic is calculated by inputting the input image data to the individual characteristic recognition module. The method of claim 2,
When a new cosmetic classification standard is added for a specific subject,
Obtaining, by the server, individual appearance characteristics of a new appearance classification criterion for the training image data, and constructing a new training data set; And
The server training a new individual feature recognition module based on the new learning data set, and adding it to the external feature recognition model. The method of claim 6,
Inputting the input image data from which the appearance description data has been obtained by the individual characteristic recognition module already established by the server into a new individual characteristic recognition module, and adding detailed information about the new appearance classification criteria; And
The method further comprising, updating the matching algorithm by matching, by a server, individual external characteristics of the new external appearance classification criteria with each abstract characteristic. The method of claim 1,
Receiving, by a server, setting data matching the combination of the abstract characteristic and the appearance classification criteria from an expert client, and setting the matching algorithm. The method of claim 1,
The step of generating the appearance description data,
Extracting code values corresponding to a plurality of individual appearance characteristics of the input image data; And
Comprising the step of combining the plurality of code values to generate appearance description data in the form of a code string. The method of claim 1,
A reference image data acquisition step, wherein the server obtains reference image data from a user client, and the reference image data is a reference for searching for an image including those having similar appearance characteristics among the same object;
Calculating individual appearance characteristics for a plurality of appearance classification criteria by inputting the reference image data to the appearance characteristic recognition model;
Generating, by a server, appearance description data by combining a plurality of individual appearance characteristics with respect to the reference image data; And
A step of extracting, by a server, image data including outline description data identical to the reference image data. The method of claim 1,
As the server receives a request for providing additional image data from a user client, sequentially providing image data having different appearance classification criteria; And
When one or more image data is selected from the additional image data by the user, the server setting a personalized abstract characteristic based on the external description data of the selected image data; further comprising, image data search method. The method of claim 11,
The method further comprising, when an abstract characteristic corresponding to the outline description data of the selected image data exists, providing, by a server, an abstract characteristic suitable for extracting the selected image data to a user client. The method of claim 1,
When the image data is moving image data including a plurality of frames,
The step of calculating the individual appearance characteristics,
It characterized in that it is performed for each frame in the video data,
The step of generating the appearance description data,
Characterized in that it is generated by sequentially listing a plurality of individual appearance characteristics for each frame,
The abstract characteristic is an expression representing a specific shape change or motion. An image search server apparatus comprising one or more computers and executing the method of any one of claims 1 to 13. An image data retrieval program combined with a computer as hardware and stored in a medium to execute the method of any one of claims 1 to 13.

Images