KR102227896B1 - System, method and program for obtaining appearance descriptive data from image data - Google Patents

Abstract

The present invention relates to a method, an apparatus, and a program for obtaining outline description data of image data.
A method of obtaining outline description data of image data according to an embodiment of the present invention includes: calculating object type information by inputting the image data into an object type recognition model by a server (S3100); Calculating, by the server, the image data into an appearance characteristic recognition model, and calculating individual appearance characteristics for a plurality of appearance classification criteria (S3200); And generating, by the server, the external description data by combining the object type information calculated for the image data and a plurality of individual external characteristics (S3300).

Description

Appearance description data acquisition method, device and program of image data {SYSTEM, METHOD AND PROGRAM FOR OBTAINING APPEARANCE DESCRIPTIVE DATA FROM IMAGE DATA}

The present invention relates to a method, an apparatus, and a program for obtaining outline description data of image data.

With the recent development of the Internet, social media network services are rapidly growing. As a result, as the amount of multimedia explosively increases, an effective image retrieval system is required, and image annotations are increasingly important due to the necessity of efficient image retrieval according to the explosively increasing web images.

Most of the image retrieval studies have been mainly conducted on a content-based image retrieval (CBIR) method that analyzes the contents of an image. Content-based image search analyzes the content of an image using visual features such as color, texture, and shape. This method works well when the number of tags to be defined is small, but performance deteriorates as the data set becomes larger and the types of tags diversify.

Text-based image retrieval (TBIR) is a method of searching for images corresponding to text by querying text. In this method, the visual content of the image is represented by a manually tagged text descriptor, and is used to perform image search in the dataset management system. That is, in the existing image or video search method, a search is performed based on information directly tagged by a user. In addition, the method of acquiring user interest information through an image is also acquired based on information directly tagged by the user, and there is a problem in that the acquisition result becomes inaccurate if the user incorrectly tags the keyword in the image. In addition, since there may be differences in keywords defined for each user, there is a problem in that the results provided according to the keywords selected by the user inputting an image are different.

Unexamined Patent Publication No. 10-2018-0133200, 2018.12.13

The present invention is to obtain type information of a specific object, external characteristics representing the external appearance of the object, or abstract characteristics recognizable for the object from image data, and generate the external description data in the form of a code string. It is intended to provide a method, apparatus, and program for obtaining appearance description data.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

According to an exemplary embodiment of the present invention, a method for obtaining outline description data of image data includes: calculating object type information by inputting the image data into an object type recognition model by a server; Calculating individual appearance characteristics for a plurality of appearance classification criteria by the server inputting the image data into an appearance characteristic recognition model; And generating, by the server, the object type information calculated with respect to the image data and a plurality of individual appearance characteristics to generate appearance description data; wherein the appearance classification criterion includes a specific object for describing the appearance of a specific object. As a classification criterion, a plurality of individual appearance characteristics expressing various external appearance characteristics within the same classification standard of the object are included.

In addition, in another embodiment, the appearance characteristic recognition model includes a plurality of individual characteristic recognition modules for determining different appearance classification criteria, and the individual characteristic recognition module includes individual appearance characteristics included in a specific appearance classification standard. And calculating a plurality of individual appearance characteristics by inputting the image data into a specialized appearance characteristic recognition model corresponding to the calculated object type information, and in the step of calculating the individual appearance characteristics, the specialized appearance characteristics The recognition model is characterized in that it is an appearance characteristic recognition model including individual characteristic recognition modules of a plurality of appearance classification criteria set in advance to be applied according to specific object type information.

In another embodiment, when it is not possible to calculate the individual appearance characteristics with respect to the external appearance classification criteria of the specialized external characteristic recognition model, the server inputs the image data to the object type recognition model again, and is different from the previous one. It further includes; calculating object type information.

In another embodiment, in the step of calculating the object type information, the server obtains a first probability and calculates type information in which the first probability is equal to or greater than a threshold value, and the type information in which the first probability is equal to or greater than a threshold value is In the case of a plurality of pieces, all of a plurality of candidate type information is calculated, the first probability is a probability that the object corresponds to each of the specific type information, and the step of calculating the individual appearance characteristics is performed with respect to the image data. When a plurality of candidate type information is calculated, each of the image data is input to a specialized appearance characteristic recognition model corresponding to each of the candidate type information.

In another embodiment, the server sets the candidate type information with the smallest appearance classification criteria for which individual appearance characteristics cannot be calculated among the specialized appearance characteristic recognition models corresponding to each of the plurality of candidate type information as the final type information. It further includes;

In addition, in another embodiment, the server calculating an abstract characteristic based on a plurality of individual appearance characteristics calculated; further comprising, the appearance description data, the abstract characteristic calculated for the image data It characterized in that it further includes.

In another embodiment, in the step of calculating the abstract characteristics, a score for each of a plurality of abstract characteristics is set for each individual appearance characteristic, and the server determines the plurality of individual appearance characteristics calculated for the image data. It is characterized by calculating by summing the scores for each set abstract characteristics.

In another embodiment, the server inputs the outline description data of the image data into a detailed type recognition model to calculate object detailed type information, wherein the detailed type information comprises: It is characterized in that it is a sub-concept of object type information.

In addition, in another embodiment, the detailed type recognition model calculates detailed type information for an individual appearance characteristic combination of the input appearance description data, as one or more individual appearance characteristic combinations are matched with one or more detailed type information. It is characterized by that.

In addition, in another embodiment, the server receiving a search keyword from a user client; And extracting, by the server, image data including appearance description data corresponding to the search keyword and transmitting the extracted image data to the user client.

In another embodiment, the generating of the appearance description data includes: extracting object type information for the image data and a code value corresponding to each of a plurality of individual appearance characteristics; And generating external description data in the form of a code string by combining the plurality of code values.

In another embodiment, when the image data is moving image data including a plurality of frames, the object type information calculation step and the individual appearance characteristic calculation step are performed for each frame in the moving image data. It is done.

A server apparatus for obtaining outline description data of image data according to another embodiment of the present invention includes one or more computers, and performs the above-mentioned method for obtaining outline description data.

A program for obtaining appearance description data of image data according to another embodiment of the present invention is combined with hardware to execute the aforementioned method for obtaining appearance description data, and is stored in a medium.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention, it is possible to obtain type information, individual appearance characteristics, or abstract characteristics of an object according to the same criterion for a plurality of image data, and generate appearance description data based on this.

In addition, according to the present invention, by calculating the object type information of specific image data and inputting the image data to a corresponding specialized appearance characteristic recognition model, it is more efficient by different combinations of appearance classification criteria applied according to the type information. Individual appearance characteristics can be obtained with this.

In addition, according to the present invention, for specific image data, abstract characteristics are calculated based on external appearance description data (obtained object type information or individual external characteristics), and the calculated abstract characteristics are added to the external description data. It is possible to acquire abstract characteristics of image data and update appearance description data.

In addition, according to the present invention, it is possible to minimize a situation in which the external feature recognition model needs to be retrained or additionally learned. For example, when a new appearance classification criterion is added for a specific object, only a new individual characteristic recognition module needs to be learned, and the existing individual characteristic recognition module can be used as it is.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a flowchart of a method of obtaining outline description data of image data according to an embodiment of the present invention.
2 is a block diagram of an external feature recognition model according to an embodiment of the present invention.
3 is an exemplary diagram for explaining a code system according to an embodiment of the present invention.
4 is a flowchart of a method of obtaining outline description data further including a step of calculating abstract characteristics according to an embodiment of the present invention.
5 is a block diagram of an abstract feature recognition model according to an embodiment of the present invention.
6 is an exemplary view for explaining setting of a first individual emotional characteristic score for an individual external characteristic according to an embodiment of the present invention.
7 is a flowchart of a method for obtaining appearance description data further including a step of calculating detailed type information according to an embodiment of the present invention.
8 is a flow chart of a method for obtaining appearance description data further including a user search step according to an embodiment of the present invention.
9 is a configuration diagram of a server for obtaining appearance description data according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and "and/or" includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

In the present specification, the'computer' includes all various devices capable of performing arithmetic processing and providing a result to a user. For example, computers are not only desktop PCs and notebooks, but also smart phones, tablet PCs, cellular phones, PCS phones, and synchronous/asynchronous systems. A mobile terminal of IMT-2000 (International Mobile Telecommunication-2000), a Palm Personal Computer (PC), a personal digital assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to a server 10 that receives a request from a client and performs information processing.

In the present specification, a'client' refers to all devices including a communication function that users can install and use a program (or application). That is, the client device may include at least one of a telecommunication device such as a smart phone, a tablet, a PDA, a laptop, a smart watch, and a smart camera, and a remote controller, but is not limited thereto.

In the present specification,'object' means an article of a specific classification or category included in image data.

In this specification,'image data' means a two-dimensional or three-dimensional static or dynamic image including a specific object. That is,'image data' may be static image data that is one frame, or dynamic image data (ie, moving image data) in which a plurality of frames are consecutive.

In the present specification, the'appearance classification standard' refers to a classification standard of appearance expression required for describing the appearance of a specific object or for annotation. That is, the'appearance classification standard' is a specific classification standard for describing the appearance of a specific object, and includes a plurality of individual appearance characteristics expressing various external characteristics within the same classification standard of the object. For example, when the object is clothing, the appearance classification standard is a classification standard for the appearance of the clothing, and may correspond to a pattern, color, fit, length, and the like. That is, when the appearance classification standard for a specific object increases, the appearance of a specific article belonging to the object can be described in detail.

In this specification,'individual appearance characteristics' refers to various characteristics included in a specific appearance classification standard. For example, when the appearance classification criterion is color, individual appearance characteristics mean various individual colors.

In the present specification, the'abstract characteristic' is an abstract characteristic perceived with respect to a specific object, and includes a first emotional characteristic or a second emotional characteristic.

In the present specification, the'first emotional characteristic' means an emotional characteristic recognized based on the appearance of a specific object. For example, it may be an emotional or trendy expression such as'cute' or'vintage' about the appearance of a specific object.

In the present specification, the'second emotional characteristic' means an emotional characteristic recognized based on information given to a product of a specific object. For example, it may be an emotional expression such as'cheap' or'expensive' perceived for a price among product information of a specific object.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart of a method of obtaining outline description data of image data according to an embodiment of the present invention.

Referring to FIG. 1, a method of obtaining outline description data of image data according to an embodiment of the present invention includes: calculating object type information by inputting image data into an object type recognition model by a server (S3100); Calculating, by the server, the image data into an appearance characteristic recognition model, and calculating individual appearance characteristics for a plurality of appearance classification criteria (S3200); And generating, by the server, the object type information calculated for the image data and a plurality of individual appearance characteristics to generate appearance description data (S3300). Hereinafter, a detailed description of each step will be described.

The server 10 calculates object type information by inputting image data into the object type recognition model 100 (S3100). That is, object type information, which is a specific classification or category of an object included in the image data, is calculated.

In an embodiment, the object type information may be calculated based on a preset object type classification. For example, the upper type classification is set to clothing, accessories, shoes, interior, etc., and the subtype classification for clothing among the upper types is'Shirt & Blouse','Jacket','Coat','Pants','Skirt' When the number of','leggings & stockings' and'one piece' is set, when image data for clothing is input, the object type recognition model may calculate object type information from among seven subtypes for clothing.

In addition, in an embodiment, in the calculating of the object type information (S3100), a plurality of types of information may be calculated. That is, when it is difficult to calculate one specific object type information with respect to the input image data, a plurality of types of information with high probability may be calculated.

In one embodiment, the object type information calculation step (S3100) may be to obtain a first probability that an object of specific image data corresponds to each specific type information, and calculate type information in which the first probability is equal to or greater than a threshold value. have.

For example, if the threshold value is 0.4, the first probability that the object of the input image data corresponds to'Shirt & Blouse' is 0.8,'Jacket' is 0.1, and'Coat' is 0.1,'Shirt & Blouse' 'Can be calculated as object type information.

In addition, in an embodiment, when there are a plurality of type information having the first probability equal to or greater than a threshold value, the plurality of type information may be calculated as candidate type information.

For example, if the threshold value is 0.4, the first probability corresponding to the'Shirt & Blouse' is 0.5, the'Jacket' is 0.4, and the'Coat' is 0.1, then'Shirt & Blouse' and'Jacket' are Can be calculated as candidate type information. The setting of the final type information for the candidate type information will be described later.

Subsequently, the server 10 inputs the image data into the appearance characteristic recognition model 200 to calculate individual appearance characteristics for a plurality of appearance classification criteria (S3200).

In one embodiment, the appearance characteristic recognition model 200 includes a plurality of individual characteristic recognition modules 210 for determining different appearance classification criteria. That is, the external characteristic recognition model includes a plurality of individual characteristic recognition modules specialized for recognizing each appearance classification criterion. The more the appearance classification criteria of a specific object are, the more the server includes a plurality of individual characteristic recognition modules in the external characteristic recognition model. The individual characteristic recognition module calculates individual appearance characteristics included in a specific appearance classification criterion of image data.

In addition, in an embodiment, the individual characteristic recognition module 210 is trained through a deep learning learning model by matching individual external characteristics of a specific external classification criterion with respect to a plurality of training image data. That is, the individual characteristic recognition module is constructed with a specific deep learning algorithm, and learning is performed by matching a specific one of a plurality of appearance classification criteria with image data for learning.

To this end, the server 10 may perform a process of training each individual characteristic recognition module 210 as follows.

First, the server acquires a plurality of training image data for a specific object. For example, when the object is a specific type of clothing (eg, a shirt), the server acquires images of several shirts.

The server defines each appearance classification standard and acquires a plurality of individual appearance characteristics for each appearance classification standard. That is, the server sets the initial number of individual characteristic recognition modules according to the setting of a plurality of appearance classification criteria. In addition, as the server sets a plurality of individual appearance characteristics in each appearance classification criterion, the server sets a type of feature to label the learning image data for each appearance classification criterion.

In an embodiment, the server may receive a plurality of appearance classification criteria for analyzing a specific object appearance and a plurality of individual appearance characteristics within each appearance classification standard from an expert client for analyzing the appearance of a specific object. For example, when constructing a model for recognizing external appearance characteristics for clothing, the server may receive external appearance classification criteria and individual appearance characteristics included therein from a client of a designer who is a clothing expert.

Thereafter, the server labels the training image data with a plurality of individual appearance characteristics of each appearance classification criterion. That is, the server receives and matches at least one individual appearance characteristic for each of the plurality of appearance classification criteria for each learning image data.

Thereafter, the server performs training on the basis of the training image data and the training data set in which the individual appearance characteristics of the specific appearance classification criteria labeled thereto are matched. That is, when the server trains the individual characteristic recognition module for the A appearance classification criterion, only the training image data and the individual appearance characteristics of the A appearance classification criterion matched therewith are extracted from the training dataset and input to the deep learning learning model. Through this, the server 10 constructs each individual characteristic recognition module 210 capable of recognizing individual external characteristics of each external classification standard.

In addition, in one embodiment, the individual appearance characteristic calculation step (S3200) includes inputting image data to each individual characteristic recognition module in the appearance characteristic recognition model, and calculating a plurality of individual appearance characteristics for the image data. will be. Through this, the server acquires all of the individual appearance characteristics of each appearance classification criterion for the image data.

In addition, in an embodiment, in the step of calculating the individual appearance characteristics (S3200), a plurality of individual appearance characteristics may be calculated by inputting the image data into a specialized appearance characteristic recognition model corresponding to the calculated object type information. .

In one embodiment, the specialized external characteristic recognition model includes individual characteristic recognition modules of a plurality of external appearance classification criteria set in advance to be applied according to specific object type information. That is, the type of the external classification criterion to be applied may be determined according to the object type information calculated for specific image data.

Specifically, referring to FIG. 2, a specialized external characteristic recognition in which a combination of different external classification criteria (i.e., a combination of individual characteristic recognition modules) is set according to specific object type information (object 1, object 2, object 3). A model can be created, and individual external characteristics are calculated by inputting the image data into each of a plurality of individual characteristic recognition modules in the specialized external characteristic recognition model of the corresponding type information according to the object type information calculated for specific image data. can do. In addition, it goes without saying that the combination of individual characteristic recognition modules in the specialized external characteristic recognition model of different object type information may be the same.

For example,'neckline','sleeve','top length', and'color' individual feature recognition modules for specialized external feature recognition models of'jacket' type information. Including, and when the specialized external characteristic recognition model of the'pants' type information includes individual characteristic recognition modules of'waist','bottom length', and'color', appearance characteristics Depending on the object type information (jacket/pants) of the image data input to the recognition model, the specialized external characteristic recognition model (that is, a combination of applied appearance classification criteria) to which the image data is input may be different.

In addition, in an embodiment, as shown in FIG. 2, each specialized external feature recognition model for a plurality of object type information may share and use a general-purpose individual feature recognition module. The general-purpose individual characteristic recognition module means an individual characteristic recognition module for a general-purpose appearance classification criterion that can be applied irrespective of the object type.

For example, external classification standards such as'color','pattern', and'texture' can be applied regardless of the type of object (individual appearance characteristics are calculated), so general-purpose appearance classification standards It may correspond to.

In addition, in an embodiment, when a plurality of candidate type information is calculated in step S3100, the image data may be respectively input to all of the specialized appearance characteristic recognition models corresponding to each of the candidate type information. The final type information setting performed based on this will be described later.

For example, when'jacket' and'pants' are calculated as candidate type information for specific image data, the image data is inputted into each of the'jacket'-specialized external feature recognition model and the'pants'-specific external feature recognition model. Individual appearance characteristics can be calculated.

In addition, although not shown in the drawings, the method for acquiring appearance description data of image data according to an embodiment of the present invention is, when it is not possible to calculate individual appearance characteristics with respect to the appearance classification criteria of the specialized appearance characteristic recognition model. And calculating, by the server, the image data into the object type recognition model again to calculate object type information different from the previous one.

That is, in one embodiment, the combination of the external classification criteria applied according to the object type information calculated as described above is different, but if an error occurs more than a certain ratio with respect to the specialized external characteristic recognition model of the calculated type information (For example, when the number of individual characteristic recognition modules that cannot calculate individual appearance characteristics is equal to or greater than a predetermined threshold), the server determines that the object type information is incorrectly calculated, and the image data is re-entered into the object type recognition model. It is to recalculate new type information.

For example, for image data related to'pants', the server incorrectly calculates the object type information as'jacket' in step S3100, and in step S3200, a'jacket'-specialized appearance characteristic recognition model (for example,'neck shape ( neckline)','sleeve length','top length', and'color' individual feature recognition modules included). If it is not possible to calculate the individual appearance characteristics for the length of the sleeve or top, the server determines that the object type information is incorrectly calculated, and the image data is re-entered into the object type recognition model to provide new type information (e.g.,'pants). ') can be recalculated.

In addition, in another embodiment, in step S3100, when a plurality of candidate type information is calculated and image data is input to each individual characteristic recognition module of a specialized external characteristic recognition model corresponding to each of the candidate type information, the individual external characteristic is determined. It further includes a step of setting the candidate type information having the smallest appearance classification criteria that cannot be calculated as the final type information.

That is, as in the above-described embodiment, it is difficult to calculate one specific object type information for the input image data, so that a plurality of candidate type information with high probability is calculated, and the image data is a specialized external characteristic recognition model for each candidate type information. When input to, candidate type information having a high recognition rate (the number of individual feature recognition modules that cannot calculate individual appearance characteristics is small) can be set as the final type information.

For example, for the video data on the jacket, candidate type information of'jacket' and'pants' is calculated in step S3100, and the corresponding video data is converted into a'jacket'-specialized appearance characteristic recognition model and a'pants' specialized appearance characteristic recognition model. In the case of each input, the recognition rate of each specialized external feature recognition model will be higher in the'jacket' than in the'pants' unless there is a special situation (the number of individual feature recognition modules that cannot calculate individual appearance characteristics is specialized in'pants'). Since there will be more in the external feature recognition model),'jacket' can be set as the final type information (ie, object type information).

Subsequently, the server generates appearance description data by combining the object type information calculated for the image data and a plurality of individual appearance characteristics (S3300). That is, if the external shape classification criteria for a specific object are divided in detail, the external shape description data specifically describes the external shape of the object through individual external characteristics.

For example, for specific image data about a shirt, the object type information calculated in step S3100 is'Shirt & Blouse', and according to a plurality of appearance classification criteria calculated in step S3200 (neck shape, sleeve, top length, color And pattern), the individual appearance characteristics of'Round Neckline','short sleeve','crop','red right' and'polka dot) In the case of', the appearance description data for the image data may be generated as "shirt & blouse, round neck, short sleeve, crop top, bright red, polka dot pattern".

In addition, in an embodiment, the step of generating the appearance description data (S3300) includes: extracting object type information for the image data and a code value corresponding to each of a plurality of individual appearance characteristics; And generating external description data in the form of a code string by combining the plurality of code values. That is, as the server codes the individual appearance characteristics, the appearance description data can be generated as a code sequence, and through this, the processing of the appearance description data can be efficiently performed.

For example, object type information (shirt & blouse) and code values corresponding to a plurality of individual appearance characteristics (round neck, short sleeves, crop top, bright red, polka dots) are respectively'Zb01','Bb01', ' In the case of Bg03', Bi01','Oa01', and'Id00', the outline description data for the image data may be generated as "Bb01, Bg03, Bi01, Id00, Oa01, Zb01" by combining the code values.

In addition, the structure of a code system of appearance description data according to an embodiment of the present invention may be variously constructed according to each object type classification standard, appearance classification standard, and abstract characteristic classification standard.

For example, a code system according to an embodiment of the present invention may be constructed as shown in FIG. 3. Referring to FIG. 3, capital letters indicate information on an upper category. That is, a code corresponding to the object type information may include Z, a code corresponding to an abstract characteristic may include A, and codes corresponding to a plurality of appearance classification criteria may include B to Y.

In addition, in the case of the code (Z) corresponding to the object type information, lowercase letters (a to y) are given according to the upper type (for example, clothing, accessories, shoes, interior, etc.), and each Numbers (00 to 99) may be assigned according to the object type information. For example, Zb may be a code value corresponding to the object type information of'Clothing', Zc is'Accessory', Zb01 is'Shirt & Blouse', and Zb02 is'Jacket'.

In addition, codes corresponding to a plurality of appearance classification criteria (B to Y), universal appearance classification standards (E, I, O, U) that can be applied irrespective of the object type, and specialized appearance classification standards that are applied to a specific object type ( B, C, D, etc.), and the code of the specialized external classification criteria may correspond to the above-described higher type codes (Za, Zb, Zc, etc.).

For example, a plurality of specialized cosmetic classification standards for'clothing' include B, and lowercase letters (a~y) are assigned according to each appearance classification standard, and numbers according to the individual appearance characteristics of the corresponding appearance classification standard (00~99) may be given. Specifically, Ba stands for'Top Silhouette', Bb stands for'Neckline', Bi stands for'Pants Silhouette', and Bb01 stands for'Round Neckline'. )'and Bb02 may be a code value corresponding to each individual appearance characteristic for the Bb appearance classification standard of the'V Neckline'. Examples of code values corresponding to abstract characteristics will be described later.

As described above, an example of the structure of a code system according to an embodiment of the present invention has been described, but the code system of the present invention is not limited thereto and may be constructed in various ways.

Referring to FIG. 4, the method for obtaining external appearance description data of image data according to an embodiment of the present invention further includes: calculating an abstract characteristic based on a plurality of individual appearance characteristics calculated by the server (S3400); do.

In one embodiment, the step of calculating the abstract characteristic (S3400) includes calculating the abstract characteristic by inputting the outline description data into the abstract characteristic recognition model 300 by the server.

That is, according to the present invention, by calculating the abstract characteristics based on the appearance description data (object type information and individual appearance characteristics) calculated from the image data, not the image data itself such as an image, efficient processing of data and objective abstract characteristics It is possible to calculate.

In addition, in one embodiment, when the abstract characteristic is calculated by inputting the outline description data in which the object type information and the individual appearance characteristics are combined into the abstract characteristic recognition model, the abstract characteristic is added to the external appearance description data to obtain the external description data. It may further include the step of updating.

In one embodiment, the abstract characteristic includes a first emotional characteristic. The first emotional characteristic is an emotional characteristic recognized based on an external shape of a specific object, and includes a plurality of first individual emotional characteristics that are specific emotional characteristics.

In one embodiment, the definition and number of each first individual emotional characteristic included in the first emotional characteristic may be set by the server, and may be added or changed. The first emotional characteristic, which is an emotional characteristic with respect to the external appearance of an object, may be defined differently according to the times or regions, and is thus to be variously changed according to the period or region.

For example, each of the first individual emotional characteristics may be set as'cute','soft','modern', etc. In addition, each first individual emotional characteristic It may be set to further include contrasted first individual emotional characteristics such as'Elegant','Wild', and'Classic'. However, the first individual emotional characteristic of the present invention is not limited thereto and may be variously set.

Referring to FIG. 5, in an embodiment, the abstract feature recognition model 300 includes a first emotional feature recognition model 310 that receives individual external features and calculates a first emotional feature.

In one embodiment, in the first emotional characteristic recognition model, as points for each of the plurality of first individual emotional characteristics are set for each individual external characteristic, each of the first individual emotions set in the input plurality of individual external characteristics. It may be to calculate the first emotional characteristic by summing the scores for the characteristic.

For example, as shown in FIG. 6, a score for each of the first individual emotional characteristics may be set for each of a plurality of individual appearance characteristics included in each appearance classification criterion. In FIG. 6, each score is displayed as 0 or 1, but the score is not limited thereto and may be set in various ways, such as a number between 0 and 1 or a negative number.

In an embodiment, a score table in which a score for each individual emotional characteristic is set for each individual appearance characteristic may generate not only one score table but also a plurality of different score tables. For example, a score table may be different for each country or region, or a personalized score table may be generated for each user, and the score table may be freely changed by the server.

Subsequently, based on the input plurality of individual external characteristics and each first individual emotional characteristic score set in the individual external characteristic, each first individual emotional characteristic score is summed to calculate a first emotional characteristic.

For example, in the example of FIG. 6, when individual appearance characteristics of'V-neck' and'red' calculated from specific image data are input, the summed score for each of the first individual emotional characteristics is'cute: 1' ,'Elegant: 2','Soft: 0','Rough: 1','Modern: 0','Classic: 1', and based on this, the first emotional characteristic can be calculated.

In an embodiment, the first emotional characteristic may be calculated by including a ratio of each first individual emotional characteristic score to the total score. For example, in the case of the above example, since the total score is 5, the first emotional characteristic is "cute: 0.2, elegant: 0.4, soft: 0, rough: 0.2, modern Classic: can be calculated as 0, classic: 0.2".

In another embodiment, the first emotional characteristic may be calculated including each first individual emotional characteristic score. For example, in the case of the above example, the first emotional trait is "cute: 1, elegant: 2, soft: 0, rough: 1, modern: 0, classic: 1, so as to include each first individual emotional attribute score. Can be calculated as ".

In another embodiment, the first emotional characteristic may be to calculate only the first individual emotional characteristic in which each first individual emotional characteristic score is equal to or greater than a preset threshold. For example, in the case where the threshold value is 2 (or a ratio of 0.4) in the above example, only the first individual emotional characteristic of'elegant' may be calculated as the first emotional characteristic. However, the calculation of the first emotional characteristic is not limited to the above-described example and may be calculated in various ways.

In addition, in an embodiment, the code value corresponding to the first emotional characteristic may include information on a score summed for each of the first individual emotional characteristics.

Specifically, in the above example, the first emotional characteristic is calculated as "cute: 0.2, elegant: 0.4, soft: 0, rough: 0.2, modern: 0, classic: 0.2", and each of the first individual emotional characteristics If the code value corresponding to "cute: Aa, elegant: Ac, soft: Ad, rough: Af, modern: Ai, classic: Ap", the external description data in the form of a code string for the first emotional characteristic is It can be produced as "Aa20, Ac40, Ad00, Af20, Ai00, Ap20". In addition, if the code values corresponding to the individual appearance characteristics'Red' and'V-Neck' are'Oa02,'Bb02', the outer form description data of the image data is a combination of "Aa20, Ac40, Ad00, Af20, Ai00, Ap20 , Bb02, Oa02". However, as described above, the code system of the present invention is not limited thereto and can be constructed in various ways.

In one embodiment, the abstract characteristic includes a second emotional characteristic. The second emotional characteristic is an emotional characteristic recognized based on information given to a product of a specific object, and includes a second individual emotional characteristic, which is a variety of emotional characteristics felt for different types of product information.

For example, a second individual sensitivity characteristic of'cheap' and'expensive' felt for product information of'price', or a second individual sensitivity characteristic of'fast' and'slow' felt for product information of'delivery time' It may include.

Referring to FIG. 5, in one embodiment, the abstract feature recognition model 300 includes a second emotional feature recognition model 320 that receives product information on an object of image data and calculates a second emotional feature. do.

In addition, in an embodiment, the second emotional characteristic recognition model 320 includes a plurality of second individual emotional characteristic recognition modules for determining emotional characteristics for different product information. The second individual emotional characteristic recognition module is configured to calculate each second individual emotional characteristic for specific product information of an object of image data.

In addition, in an embodiment, the second emotional characteristic may be calculated in consideration of not only the product information, but also various information such as individual appearance characteristics, object type information, or user information.

Referring to FIG. 7, in a method for obtaining outline description data of image data according to an embodiment of the present invention, the server 10 inputs outline description data of the image data into a detailed type recognition model 400, It further includes; calculating detailed type information (S3500).

In an embodiment, the detailed type information may be a sub-concept of the object type information calculated in step S3100. For example, when object type information of'pants' is calculated in step S3100 for specific image data, detailed type information may be'skinny pants','slacks pants','exhaust pants', and the like.

The detailed type recognition model may be a model that calculates object detailed type information of the image data when external appearance description data (object type information, individual appearance characteristics) generated for specific image data is input.

In one embodiment, the detailed type recognition model may be to calculate detailed type information for an individual external characteristic combination of the input external appearance description data as one or more detailed type information is matched to each of one or more individual external characteristic combinations. have. In addition, in one embodiment, it is also possible to calculate a plurality of detailed type information.

For example, detailed type information (a2 which is a sub-concept when the object type information is A, b1 which is a sub-concept when the object type information is B) is matched with respect to a specific combination of appearance characteristics (a, b, c). In addition, it is possible to calculate the corresponding detailed type information (b1) according to the input appearance description data (object type information B, individual appearance characteristics a, b, c).

In addition, in an embodiment, when the detailed type information is calculated, the appearance description data may further include the object detailed type information. That is, in an embodiment, a code value corresponding to the calculated detailed type information may be extracted and added to the existing appearance description data.

Referring to FIG. 8, a method of obtaining external description data of image data according to an embodiment of the present invention includes: receiving, by the server, a search keyword from a user client (S3600); And extracting, by the server, image data including appearance description data corresponding to the search keyword and transmitting the extracted image data to the user client (S3700). Hereinafter, a detailed description of each step will be described.

That is, when a user inputs a specific search keyword to search for image data, the server extracts object type information, individual appearance characteristics, or abstract characteristics corresponding to the search keyword, and extracts type information, individual appearance characteristics, or abstract characteristics. Image data including characteristic appearance description data is provided to the user as a search result.

For example, if a user enters'cute red round T-shirt' as a search keyword,'Shirt & Blouse' is used as object type information from the search keyword, and'red','round neck','color' as individual appearance characteristics. It is possible to extract'no (collar)' and'cute' as an abstract characteristic, and provide image data including appearance descriptive data of "red, round neck, no color, cute" to the user.

In addition, in an embodiment, when the image data corresponding to the search keyword is extracted, preference information for each of the user's individual appearance characteristics, first individual sensibility characteristics, or second individual sensibility characteristics may be further considered and extracted. . In this case, even when the same search keyword is input, the search result may be different depending on the user who inputs the search keyword.

In addition, although not shown in the drawings, in the method for obtaining appearance description data of image data according to an embodiment of the present invention, when a new appearance classification criterion for a specific object is added, the server classifies the new appearance of the image data for learning. Acquiring the individual appearance characteristics of the reference, and constructing a new learning data set; And training, by the server, a new individual feature recognition module based on the new learning data set, and adding it to the external feature recognition model.

That is, when a new appearance classification criterion for a specific object is added (for example, a new criterion for dividing the appearance characteristics of clothing is added), the server does not change the existing individual characteristic recognition module. It is possible to change the external characteristic recognition model to suit the situation in which a new external classification standard has been added by additionally building only individual characteristic recognition modules.

First, the server acquires the individual appearance characteristics of the new appearance classification criteria for the training image data, and constructs a new training data set. In one embodiment, in the case of constructing a new individual characteristic recognition module by using the same image data used to train other individual characteristic recognition modules, the server is based on a new appearance classification standard for each of the image data for learning from an expert client. The individual appearance characteristics of are inputted. In addition, in another embodiment, the server acquires new image data for training the individual characteristic recognition module for the new appearance classification criteria, and receives the individual appearance characteristics of the new appearance classification criteria for this, and receives the new learning data set. Build.

Thereafter, the server trains a new individual feature recognition module based on the new learning data set, and adds it to the external feature recognition model. Through this, the server adds a new individual characteristic recognition module along with a plurality of existing individual characteristic recognition modules to the existing external appearance characteristic recognition model (multiple specialized external characteristic recognition models).

In addition, in an embodiment, the server inputs the image data obtained by the external appearance description data by the individual characteristic recognition module already built into the new individual characteristic recognition module to add individual external characteristics to the new external appearance classification criteria; It may contain more. That is, the server performs a process of updating the appearance description data of the previously acquired image data to reflect the new appearance classification criteria. To this end, the server performs a process of calculating individual appearance characteristics by inserting all image data into a new individual characteristic recognition module.

In addition, in an embodiment of the present invention, when the image data is moving image data including a plurality of frames, the object type information calculating step (S3100) and the individual appearance characteristic calculating step (S3200) may be performed in each of the moving picture data. It may be characterized in that it is performed on the frame of, and the appearance description data generation step (S3300) may be characterized in that the object type information for each frame and a plurality of individual appearance characteristics are sequentially arranged and generated.

A server apparatus for obtaining outline description data of image data according to another embodiment of the present invention includes one or more computers, and performs the above-mentioned method for obtaining outline description data.

Referring to FIG. 9, the server apparatus 10 for obtaining external description data of image data according to an embodiment of the present invention includes an object type recognition model 100, an external characteristic recognition model 200, and an external description data generation unit ( 600) and a database 800, and performs the above-described method of obtaining appearance description data.

In addition, in another embodiment, the server device 10 further includes one or more of an abstract feature recognition model 300, a detailed type recognition model 400, a style recognition model 500, or a recommended image data generation unit 700. Can include.

In addition, the above-described method for obtaining outline description data of image data according to an embodiment of the present invention may be implemented as a program (or application) to be executed by being combined with a computer, which is hardware, and stored in a medium.

The above-described program is C, C++, JAVA, machine language, etc. that can be read by the computer's processor (CPU) through the device interface of the computer in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in the computer language of. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server 10 at a remote location in order to execute the functions, the code can be used with any other computer remotely using the communication module of the computer. It may further include a communication-related code for how to communicate with the server 10 or the like, and what information or media should be transmitted/received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers 10 to which the computer can access, or on various recording media on the computer of the user. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You will be able to understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

Claims (14)

Calculating first object type information by inputting the image data into the object type recognition model by the server;
The server inputs the image data into a specialized appearance characteristic recognition model corresponding to the first object type information, and calculates individual appearance characteristics for a plurality of appearance classification criteria-The individual appearance characteristics determine the appearance of the object. Expressing a plurality of external characteristics within the external classification standard, which is a specific classification standard for description -; And
If the specialized external characteristic recognition model cannot calculate individual external characteristics, the server inputs the image data into the object type recognition model, and calculates second object type information different from the first object type information -The specialized external characteristic recognition model includes a plurality of individual characteristic recognition modules corresponding to a plurality of external appearance classification criteria set to be applied to specific object type information, and the individual characteristic recognition module is Including a module for calculating the characteristic -; Including,
If the specialized external characteristic recognition model cannot calculate individual external characteristics, the number or ratio of individual characteristic recognition modules for which individual external characteristics are not calculated among the plurality of individual characteristic recognition modules included in the specialized external characteristic recognition model. Is above this threshold,
In the calculating of the first object type information, the server obtains a first probability and calculates type information having the first probability equal to or greater than a threshold value, Calculate all of the plurality of candidate type information,
The first probability is a probability that the object corresponds to each specific type information,
In the calculating of the individual appearance characteristics, when a plurality of candidate type information is calculated for the image data, inputting the image data to each of a plurality of candidate appearance characteristics recognition models,
The candidate appearance characteristic recognition model is a specialized appearance characteristic recognition model corresponding to the candidate type information,
The server sets at least one candidate type information among the plurality of candidate type information as final type information,
The final type information is candidate type information corresponding to a candidate appearance characteristic recognition model having the least number or ratio of individual characteristic recognition modules for which individual appearance characteristics are not calculated. The method of claim 1,
The server further comprising the step of generating the appearance description data by combining the object type information calculated with respect to the image data and a plurality of individual appearance characteristics, the appearance description data acquisition method of image data. delete delete delete The method of claim 2,
The server further comprises the step of calculating abstract characteristics based on the calculated plurality of individual appearance characteristics,
The outline description data further includes the abstract characteristic calculated with respect to the image data. The method of claim 6,
The step of calculating the abstract characteristics,
A score for each of a plurality of abstract characteristics is set for each individual appearance characteristic, and the server calculates by summing the scores for each abstract characteristic set in a plurality of individual appearance characteristics calculated for the image data. , Method for obtaining external description data of image data. The method of claim 2,
The server further comprises the step of calculating the object detailed type information by inputting the outline description data of the image data into the detailed type recognition model,
The detailed type information is a sub-concept of the calculated first object type information or second object type information. The method of claim 8,
The detailed type recognition model,
According to one or more combinations of individual appearance characteristics matched with the one or more detailed type information, to calculate the detailed type information for the individual appearance characteristics combination of the input appearance description data, the appearance description data acquisition method of image data. The method of claim 2,
Receiving, by the server, a search keyword from a user client; And
The method further comprising the step of extracting, by the server, image data including appearance description data corresponding to the search keyword and transmitting the extracted image data to the user client. The method of claim 2,
The step of generating the appearance description data,
Extracting object type information for the image data and a code value corresponding to each of a plurality of individual appearance characteristics; And
And generating outline description data in the form of a code string by combining the plurality of code values. The method of claim 1,
When the image data is moving image data including a plurality of frames,
The first object type information calculation step, the second object type information calculation step, and the individual appearance characteristic calculation step,
A method of obtaining outline description data of image data, characterized in that it is performed for each frame in the moving picture data. A server apparatus comprising one or more computers, and executing the method of claim 1, for obtaining appearance description data of image data. A program for obtaining appearance description data of image data, which is combined with a computer as hardware and stored in a medium to execute the method of claim 1.

Images