KR20210109927A - System, method and program of constructing dataset for training visual characteristic recognition model - Google Patents

Abstract

Provided are a method, apparatus, and program for constructing a dataset for learning a visual characteristic recognition model. In accordance with an embodiment of the present invention, the method for constructing the dataset for learning the visual characteristic recognition model comprises: a learning image data acquisition step (S100) in which a server acquires a plurality of pieces of learning image data; a learning image data transmission step (S200) in which the server transmits the plurality of pieces of learning image data to an expert client; an individual characteristic reception step (S300) in which the server receives an individual characteristic on visual properties of each piece of the learning image data from the expert client; and a step (S400) in which the server constructs a learning dataset based on the plurality of individual characteristics received by the server for each piece of the learning image data. The present invention aims to provide the method, apparatus, and program for constructing a dataset for learning a visual characteristic recognition model, which is capable of more precisely learning the visual characteristic recognition model.

Description

Dataset construction method, device and program for learning visual characteristics recognition model

The present invention relates to a method, apparatus and program for constructing a dataset for learning a visual characteristic recognition model.

Recently, with the development of the Internet, social media network services are rapidly growing. As a result, as the amount of multimedia explosively increases, an effective image search system is required, and the importance of image annotation is increasing due to the need for efficient image search according to the explosively increasing web image.

Most of the image retrieval studies have mainly conducted a content-based image retrieval (CBIR) method that analyzes the contents of an image. Content-based image retrieval analyzes the content of an image using visual features such as color, texture, and shape. This method works well when the number of tags to be defined is small, but the performance deteriorates as the dataset grows and the types of tags diversify.

Text-based image retrieval (TBIR) is a method of searching for images corresponding to text by using text as a query. In this method, the visual content of an image is expressed by a manually tagged text descriptor, which is used to perform image search in a dataset management system. That is, in the existing image or video search method, a search is performed based on information directly tagged by a user. Accordingly, there is a problem in that the acquisition result is inaccurate if the user tags the keyword incorrectly on the image. Also, since there may be differences in keywords defined for each user, there is a problem in that results provided according to keywords selected by a user inputting an image are different.

Laid-open Patent Publication No. 10-2018-0133200, 2018.12.13

An object of the present invention is to provide a method, apparatus and program for constructing a learning dataset for learning a visual characteristic recognition model capable of accurately outputting individual characteristics describing visual characteristics with respect to an input image (image data).

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method for constructing a dataset for learning a visual characteristic recognition model according to an embodiment of the present invention includes: acquiring, by a server, a plurality of image data for training including one or more objects; acquiring image data for training; The server transmits the training image data to a plurality of expert clients according to the image type of each training image data, wherein the expert client is a client of an expert who is in charge of classifying each visual attribute in the visual attribute combination, learning image data transmission step; an individual characteristic receiving step of receiving, by the server, individual characteristics for the visual attributes of each training image data from the expert client; and generating, by the server, external descriptive data for each training image data based on a plurality of individual characteristics received for each training image data, and constructing a training dataset, wherein the visual properties include, As a specific classification criterion for describing the visual characteristics of the object, it includes a plurality of individual characteristics expressing various visual characteristics within the same classification criterion, and the visual attribute combination is a plurality of viewpoints to be calculated for a specific image type. The property is preset, the training dataset is used for learning one or more visual characteristic recognition models, and the visual characteristic recognition model includes a plurality of training image data included in the training dataset and the appearance corresponding thereto. It is learned using one or more individual characteristics extracted from descriptive data.

Further, in another embodiment, the visual characteristic recognition model includes a plurality of individual attribute recognition modules for determining different visual attributes, and the plurality of individual attribute recognition modules include individual individual attribute recognition modules corresponding to each individual attribute recognition module. It is learned based on a training dataset, and the individual training dataset is one in which individual characteristics for a specific visual attribute extracted from external description data are matched to a plurality of training image data.

In another embodiment, the combination of the visual attributes is characterized in that it is combined into a plurality of detailed visual attributes that are the lowest visual attributes belonging to each visual attribute.

Further, in another embodiment, the outline description data is the outline description data in the form of a code string obtained by extracting and combining code values corresponding to a plurality of individual characteristics of the image data for training.

Also, according to another embodiment, the code value corresponding to the detailed individual characteristic includes individual characteristic information on one or more upper visual attributes of the detailed visual attribute.

In addition, in another embodiment, when a new visual attribute is added, the server transmits a plurality of training image data included in the training dataset to a client of an expert who is in charge of classifying the added new visual attribute. image data additional transmission step; receiving, by the server, a new individual characteristic for each new visual attribute of each training image data from the expert client; and extracting code values corresponding to the plurality of new individual characteristics received by the server and adding the extracted code values to the outline description data of each image data for training.

Also, in another embodiment, the method further comprises the step of adding, by the server, the new visual attribute to a visual attribute combination for one or more image types, wherein the additional transmitting image data for training includes: It is characterized in that only image data for learning of the image type is transmitted.

In addition, in another embodiment, the visual attribute includes a dynamic visual attribute for describing the movement of a specific object, and when the image data is moving image data including a plurality of frames, the step of constructing a dataset for learning includes: , characterized in that it is constructed by acquiring individual characteristics for each frame in the moving picture data and acquiring dynamic individual characteristics for a plurality of frames.

The apparatus for constructing a dataset for learning a visual characteristic recognition model according to another embodiment of the present invention includes one or more computers and performs the above-mentioned method for constructing a dataset.

A data set building program for learning a visual characteristic recognition model according to another embodiment of the present invention is combined with hardware to execute the above-mentioned data set building method, and is stored in a medium.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, by allowing the same expert to determine specific visual attributes for a plurality of learning image data, it is possible to more accurately learn a visual characteristic recognition model by constructing a learning dataset based on a unified standard.

In addition, according to the present invention, by setting the type of visual attribute applied to a specific image type in advance, and transmitting the training image data of the corresponding image type only to the set visual attribute expert client to receive individual characteristics, a training dataset It has the effect of increasing the construction efficiency.

In addition, according to the present invention, when a new visual attribute is added, a code value corresponding to the new individual characteristic is added to the external description data of a plurality of image data for learning included in the previously built learning dataset for simple learning. You can update the dataset.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a flowchart of a method for constructing a training dataset according to an embodiment of the present invention.
2 is a block diagram of a data set construction system for learning according to an embodiment of the present invention.
3 is a block diagram of a visual characteristic recognition model according to an embodiment of the present invention.
4 is a flowchart of a method of constructing a data set for learning when a new visual attribute is added according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

As used herein, the term 'computer' includes various devices capable of providing a result to a user by performing arithmetic processing. For example, computers include desktop PCs and notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. Also, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

In this specification, a 'client' refers to any device including a communication function that users can use by installing a program (or application). That is, the client device may include one or more of a telecommunication device such as a smart phone, a tablet, a PDA, a laptop, a smart watch, a smart camera, and a remote controller, but is not limited thereto.

As used herein, 'image data' refers to a two-dimensional or three-dimensional static or dynamic image including one or more objects. That is, 'image data' may be static image data that is one frame, or dynamic image data (ie, moving image data) in which a plurality of frames are continuous.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart of a method of constructing a training dataset according to an embodiment of the present invention.

Referring to FIG. 1 , a method for constructing a dataset for learning a visual characteristic recognition model according to an embodiment of the present invention includes: acquiring, by a server, a plurality of image data for training, acquiring image data for training (S100); The server transmits a plurality of training image data to the expert client, training image data transmission step (S200); an individual characteristic receiving step (S300) in which the server receives individual characteristics for the visual attributes of each training image data from the expert client; and constructing, by the server, a training dataset based on a plurality of individual characteristics received for each training image data (S400). Hereinafter, a detailed description of each step will be described.

The server 10 acquires a plurality of learning image data (S100). The image data for learning means image data used for training of a learning model.

In an embodiment, the acquiring of the image data for training includes the server acquiring the image data for training from an image provider client. The image provider client means a client of the image provider that uploads specific image data to the server.

In an embodiment, the acquisition of the image data for training includes uploading specific image data to a server.

For example, when an image of a product for sale is uploaded to a shopping mall, the image provider client is a client of a person who uploads the image to the shopping mall, and the image of the product for sale may be image data for learning.

In one embodiment, the learning model is a visual characteristic recognition model 100 . Hereinafter, a visual characteristic recognition model 100 according to an embodiment of the present invention will be described with reference to FIGS. 2 and 3 .

In an embodiment, the visual characteristic recognition model 100 is a learning model that outputs individual characteristics of a plurality of visual attributes of the image data when specific image data is input.

The visual attribute is a specific classification criterion for describing or annotating a visual characteristic of a specific object, and includes a plurality of individual characteristics expressing various visual characteristics within the same classification criterion.

For example, when the object is clothing, the visual attribute is a classification criterion for visual characteristics of clothing, and may correspond to a pattern, a color, a fit, a length, and the like. If is a person, the visual attribute may correspond to a posture, an arm motion, a hand shape, and the like.

In an embodiment, the visual attribute includes a dynamic visual attribute for describing a motion of a moving object such as 'arm motion' as well as a visual attribute for describing a visual characteristic such as an appearance of a stationary object.

Also, according to an embodiment, the visual attribute may include a specific classification criterion for simultaneously describing the visual characteristics of a plurality of objects. For example, in the case of the 'surgical operation' visual attribute, individual characteristics (which type of surgical operation) may be calculated according to the type of each object of 'surgical tool' and 'organ (surgery target)'.

Also, in one embodiment, the visual characteristic recognition model 100 includes a plurality of individual attribute recognition modules 110 for determining different visual attributes as shown in FIG. 2 . That is, the visual characteristic recognition model 100 includes a plurality of individual attribute recognition modules 110 specialized for recognizing each visual attribute. As the number of visual attributes increases, the server 10 includes a plurality of individual attribute recognition modules 110 in the visual attribute recognition model 100 . The individual attribute recognition module 110 calculates individual characteristics included in a specific visual attribute.

In addition, in one embodiment, the visual characteristic recognition model 100 includes a plurality of specialized visual characteristic recognition models 120 for recognizing visual characteristics with respect to image data of a specific image type as shown in FIG. 3 .

In one embodiment, each specialized visual characteristic recognition model 120 includes a combination of different individual attribute recognition modules 110 for each image type.

Specifically, since the visual characteristics to be calculated from the image data may be different depending on the type of image data, a visual attribute combination that is a combination of a plurality of visual attributes applied to the image data of each image type is set.

Then, by combining a plurality of individual attribute recognition modules that determine each visual attribute in the visual attribute combination for a specific image type, a specialized visual characteristic recognition model for each image type is generated.

For example, for a plurality of image types (clothing image, surgery image, exercise image), 'color', 'pattern', 'shoulder shape', and 'sleeve length' visual attributes are in the clothing image type, and the surgical image type is The visual attribute combination is set for 'surgery tool', 'organ', 'movement of surgical tool', and visual attribute combination of 'exercise equipment', 'arm and leg angle', and 'finger shape' for the motion image type. and a specialized visual characteristic recognition model for each image type can be created based on this.

Also, although not shown in the drawings, in an embodiment, the specialized visual characteristic recognition model for each image type may include a specialized visual characteristic recognition model for a plurality of subtypes within a specific image type.

For example, since visual characteristics to be calculated may be different depending on subtypes within an image type such as a top and bottom image for a clothing image or a baseball or soccer image for an exercise image, individual attributes applied to each subtype A detailed type-specific visual characteristic recognition model composed of recognition modules can be created.

In an embodiment, a plurality of visual attributes and a plurality of individual characteristics for each visual attribute may be set by the server. That is, the server sets a visual attribute, which is a criterion for judging the visual characteristics of the image data for learning, and an individual characteristic that is a type of feature to label the image data for learning for each visual attribute.

In one embodiment, the server, from an expert client for image data analysis of a specific image type, includes a plurality of visual attributes (ie, a combination of visual attributes) for describing the visual characteristics of image data of a corresponding image type and a plurality of visual attributes within each visual attribute. Individual characteristics can be set.

For example, when constructing the specialized visual characteristic recognition model 120 for a clothing image type, the server receives and sets appropriate visual attributes and individual characteristics included therein from a client of a designer who is a clothing expert, and sets each visual attribute. It is possible to construct a specialized visual characteristic recognition model 120 for an image type of clothing including an individual attribute recognition module 110 and a plurality of individual attribute recognition modules.

Next, the server transmits a plurality of learning image data to the expert client (S200).

The expert client refers to a client of an expert who performs a role of imparting individual characteristics to the image data for training (ie, labeling the image data for training).

In an embodiment, the expert client refers to a client of an expert who classifies each visual attribute in charge. That is, the individual characteristic acquisition (given) according to a specific visual attribute may be performed by the same expert.

For example, acquisition of individual characteristics of a plurality of image data for learning with respect to the visual attribute of 'color' may be performed by an expert in charge of 'color'. Since the judgment of some visual attributes can be somewhat subjective, by having the same expert judge specific visual attributes for a plurality of learning image data, it is possible to more accurately learn the visual characteristic recognition model by building a learning dataset with a unified standard. there is an effect

In one embodiment, the step of transmitting the training image data ( S200 ) may be to transmit all the training image data acquired by the server to all the expert clients. That is, all the acquired image data for learning regardless of the image type is transmitted to the expert client in charge of a specific visual attribute.

In this case, the expert client in charge of visual attributes that is not applied to a specific image type may transmit information about this to the server.

For example, if the training image data for the clothing image type is transmitted to the 'surgical tool' visual attribute expert client, information on “the corresponding visual attribute cannot be applied (individual characteristics cannot be determined)” can be transmitted to the server.

In another embodiment, the step of transmitting the image data for training ( S200 ) is to the expert client in charge of each visual attribute (ie, each visual attribute included in the visual attribute combination) set to be applied to the image type of the image data for training. The training image data may be transmitted.

Specifically, referring to the above example, the clothing image is transmitted to the expert clients in charge of each of 'color', 'pattern', 'shoulder shape', and 'sleeve length', and the surgical image is 'surgery tool' and 'organ' , the 'movement of the surgical tool' can be transmitted to the respective specialist in charge of the client.

Also, in an embodiment, the visual attribute includes a special visual attribute applied only to a specific type of object and a general visual attribute applied to all types of objects. Hereinafter, the clothing object in the clothing image will be described as an example.

For example, a visual attribute such as 'color' or 'texture' can be applied (individual characteristic calculation) to all clothing objects, and thus is a general-purpose visual attribute. In contrast, the 'neck shape' visual attribute is a specialized visual attribute for 'top' because it can be applied only to the 'top of clothing' type object.

In one embodiment, although not shown in the drawing, the method further includes: obtaining, by the server, object type information of the image data for training acquired, wherein the step of transmitting the image data for training includes the server in charge of general-purpose visual attributes and classifying This includes transmitting all the acquired image data for learning to the client of the expert, but transmitting only the training image data in which the type of object to which the specialized visual attribute is applied appears to the client of the expert who is in charge of classifying the specialized visual attribute. Hereinafter, it will be described in detail.

The server acquires object type information of the image data for learning. That is, type information of one or more objects appearing in the image data for learning is acquired.

In an embodiment, the object type information includes upper type information or lower type information of a specific object. For example, in the case of type information on 'shirt', it may be 'clothing-top-shirt' or 'shirt'. Also, in this case, 'shirt', which is the lower type information, may include information about 'clothes-top', which is the upper type information.

In one embodiment, when the specific visual attribute relates to the type of the object, such as 'object type' or 'clothing type', that is, the classification criterion for classifying the type of the object belongs to a sub-concept of the visual attribute. In the case of , acquisition of object type information may be acquisition of individual characteristics for a corresponding visual attribute (type classification criterion).

In another embodiment, the obtaining of the object type information includes obtaining, by the server, from a type classification expert client or through a separate type recognition model. However, the method of acquiring object type information is not limited to the above-described embodiment.

That is, the type classification criterion for classifying the type of the object may be a classification criterion belonging to a sub-concept of the visual attribute or a separate classification criterion independent of the visual attribute.

In one embodiment, in the step of transmitting the image data for training, the server transmits all the acquired image data for training to the client of the expert who classifies in charge of the general visual attribute, but corresponds to the client of the expert who classifies in charge of the specialized visual attribute Only image data for learning in which the type of object to which the specialized visual attribute is applied appears is transmitted.

That is, in one embodiment, based on the acquired object type information, only image data for training in which an object of a type to which the specialized visual attribute can be applied appears is transmitted to the expert client in charge of the specialized visual attribute, thereby minimizing unnecessary data transmission. The efficiency of data set construction can be improved.

For example, in the case of a general-purpose visual attribute that is applied regardless of the type of object, such as 'color', in one embodiment, the server transmits all acquired image data for clothing type learning to the expert client in charge of 'color'. On the other hand, since 'neckline' is a specialized visual attribute that is applied only to the 'top' type object, the server uses only the image data for learning the clothing type in which 'top' appears based on the acquired type information. By transmitting to the expert client in charge, it is possible to reduce the transmission of unnecessary training image data and improve the efficiency of building a training dataset. Although the specialized and general-purpose visual properties have been described using the clothing object in the clothing image as an example, the present invention is not limited thereto, and various types of objects (surgery tools, organs, etc. , exercise equipment, hands, etc.) can be applied, of course.

Next, the server receives individual characteristics of the visual attributes of each image data for learning from the expert client (S300).

In an embodiment, the server receives individual characteristics of the image data for learning for the corresponding visual attribute from the expert client in charge of the specific visual attribute.

For example, when the server transmits specific training image data for a top to a plurality of expert clients, individual characteristics of 'clothes' from the expert client in charge of 'object type' and 'shirt' individual characteristics from the expert client in charge of 'clothing type' Characteristics and 'sleeve length' individual characteristics may be received from the expert client responsible for 'sleeve length', respectively.

In an embodiment, in the individual characteristic receiving step ( S300 ), individual characteristics may be received for each object appearing in the image data, or individual characteristics of a plurality of objects may be received according to the characteristics of the visual attribute. . That is, in one visual attribute applied to one learning image data, one or a plurality of individual characteristics may be obtained according to the characteristic of the visual attribute.

In an embodiment, the individual characteristic receiving step ( S300 ) may be that the server receives the detailed individual characteristic of the image data for learning for the detailed visual attribute from the expert client in charge of the detailed visual attribute. That is, the visual attribute combination set for a specific image type is combined with detailed visual attributes, and accordingly, the image data for learning may be transmitted only to the expert client in charge of the detailed visual attribute.

The detailed visual attribute means the lowest visual attribute belonging to each visual attribute. For example, in the case of 'clothing (object type)-shirt (clothing type)-sleeveless (sleeve length)', 'sleeve length' may be a detailed visual attribute, and 'sleeveless' may be a detailed individual characteristic.

The lower visual attribute is a relatively lower visual attribute in relation to the upper visual attribute. In an embodiment, the relationship between the visual attributes may be set or changed by the server. Also, one or more lower visual attributes may exist for one upper visual attribute.

In an embodiment, the detailed individual characteristic may include individual characteristic information on one or more upper visual attributes of the detailed visual attribute to which the detailed individual characteristic belongs.

For example, 'sleeveless', which is a detailed individual characteristic, may include information about 'clothes' or 'top', which is an individual characteristic of an upper visual attribute of 'sleeve length', which is a detailed visual attribute. That is, even if the server receives only 'sleeveless', which is a detailed individual characteristic, information on 'clothes' or 'top', which is an individual characteristic for an upper visual attribute, can be obtained.

In an embodiment, the detailed visual attribute may be changed as a lower visual attribute of the detailed visual attribute is added. For example, when 'neck shape' is additionally set as a sub-visual property of the 'clothing type' visual attribute ('object type (clothing)-clothing type (shirt)-{neck shape (V-neck), sleeve length (sleeveless) }'), 'neck shape' and 'sleeve length' may be detailed visual attributes. In addition, the visual attribute including the detailed visual attribute can be freely set and changed by the server.

That is, as the server transmits training image data only to the expert clients in charge of a plurality of detailed visual attributes except for the upper visual attributes and receives detailed individual characteristics, by transmitting the training image data to the minimum expert clients, the efficiency of building a training dataset is improved. has the effect of increasing it.

For example, if specific training image data is transmitted only to the expert client in charge of 'neck shape' and 'sleeve length', which are detailed visual attributes, and 'V-neck' and 'sleeveless' are received as detailed individual characteristics, the detailed individual characteristics are Since it includes information on 'clothes' and 'tops', which are individual characteristics of upper visual attributes, it is possible to collect sufficient visual characteristics.

Next, the server builds a training dataset based on a plurality of individual characteristics received for each training image data (S400).

In one embodiment, the training dataset is for training the visual characteristic recognition model 100 , in which individual characteristics according to a plurality of visual attributes labeled in the training image data are matched.

Also, in one embodiment, the training dataset may include a plurality of individual training datasets for training each individual attribute recognition module 110 . In the individual training dataset, individual characteristics of a specific visual attribute (visual attribute determined by the individual attribute recognition module for training) labeled in the image data for training are matched.

Specifically, when the server intends to train the individual characteristic recognition module 110 for the A visual attribute, the individual learning dataset for the A visual attribute in the training dataset (only the individual characteristics of the A visual attribute in each training image data are The matched dataset) is extracted and input to the deep learning learning model. Through this, the server can train and build each individual attribute recognition module 110 capable of recognizing individual characteristics of each visual attribute.

In addition, in an embodiment, in the case of a visual attribute that is a specific classification criterion for describing the visual characteristics of a plurality of objects, the individual training dataset for the visual attribute may be one in which a plurality of individual characteristics are matched to the training image data. . For example, the individual training dataset for the 'surgery motion' visual attribute may be one in which individual characteristics for the 'surgical tool' and 'organ (surgery target)' visual attributes are matched to the training image data at the same time.

In one embodiment, when the amount of the individual training dataset for a specific visual attribute is insufficient, by additionally acquiring training image data of the image type to which the visual attribute is applied, transmitting it to the expert client in charge of the visual attribute, and receiving the individual characteristic , it is possible to construct a dataset for individual learning for the visual attribute.

In another embodiment, the training dataset may be to construct a training dataset by generating external description data based on a plurality of individual characteristics received with respect to the training image data.

That is, a training dataset is constructed by generating external description data by combining individual characteristics of a plurality of visual attributes received for specific training image data. to extract the dataset.

Specifically, when the server intends to train the individual attribute recognition module for visual attribute A, a training dataset (appearance descriptive data combining a plurality of individual characteristics for visual attributes A, B, and C for each training image data) It is possible to train the individual attribute recognition module by extracting the individual learning dataset in which only the individual characteristics of the visual attribute A are matched to the learning image data.

Hereinafter, a clothing image will be described in detail with respect to the appearance description data according to an embodiment of the present invention.

In an embodiment, the outline description data includes a combination of a plurality of individual characteristics of the image data for training. For example, "shirt (type of clothing), light pink (color), floral (pattern), slim (Top Silhouette), V-neck (neck shape), sleeveless (sleeve length)" can be combined. However, the form of the outline description data is not limited thereto, and includes all forms in which the obtained individual characteristics are combined.

In an embodiment, the outline description data includes outline description data in the form of a code string obtained by extracting and combining code values corresponding to a plurality of individual characteristics of the image data for training. That is, as individual characteristics are coded, the outline description data can be generated in the form of a code string, and through this, the outline description data can be processed efficiently.

For example, if the code value corresponding to the acquired individual characteristic is ''shirt-Zb01, light pink-Ob01, floral pattern-Ie01, slim-Ba01, V-neck-Bb02, sleeveless-Bg01”, the The outline description data may be “Ba01, Bb02, Bg01, Ie01, Ob01, Zb01”.

Referring to FIG. 2 , in one embodiment, the server receives a plurality of individual characteristics for specific training image data from an expert client, and extracts and combines code values corresponding to the individual characteristics to form external description data in the form of a code string. can create

In another embodiment, the server may receive code values corresponding to a plurality of individual characteristics of specific training image data from the expert client, and may generate the outline description data in the form of a code string by combining the plurality of code values.

Also, in an embodiment, the code value corresponding to the lower individual characteristic includes upper individual characteristic information on one or more upper visual attributes of the lower visual attribute to which the lower individual characteristic belongs.

For example, in the code value 'Bb02' corresponding to 'V-neck', “B” may include information on “clothing,” which is an individual characteristic of a higher visual attribute (object type).

4 is a flowchart of a data set construction method when a new visual attribute is added according to an embodiment of the present invention. Referring to FIG. 4 , in one embodiment, when a new visual attribute is added, the server transmits a plurality of training image data included in the training dataset to a client of an expert who is in charge of classifying the added new visual attribute A further transmission step of video data for training (S500); receiving, by the server, a new individual characteristic for each new visual attribute of each training image data from the expert client (S600); and extracting code values corresponding to the plurality of new individual characteristics received by the server and adding the extracted code values to the outline description data of each image data for learning (S700).

The server transmits the plurality of training image data included in the training dataset to the expert client who is in charge of classifying the added new visual attribute (S500). That is, in one embodiment, when a new visual attribute is additionally set and a new individual learning dataset for training a new individual attribute recognition module is required, the new visual attribute expert client can be sent to Alternatively, it is also possible to transmit newly acquired learning image data.

In one embodiment, the step S500 is to transmit only the training image data of the image type to which the new visual attribute is applied to the expert client. That is, when a new visual attribute for a visual characteristic to be additionally acquired for a specific image type is added, image data of the corresponding image type is extracted and transmitted.

For example, when it is desired to further acquire individual characteristics of the visual attribute of 'shoulder' with respect to an image of a clothing type, only image data for training of a clothing type is transmitted among a plurality of image data for training included in the training dataset. In addition, unnecessary data transmission can be minimized by not transmitting surgical and exercise images.

In another embodiment, the method may further include determining an image type to which a newly added new visual attribute is to be applied. That is, when the new visual attribute can be applied to an image type other than the originally intended image type, the image type to be applied is determined.

In another embodiment, when the added new visual attribute is a lower visual attribute, the step of transmitting the additional image data for training includes, by the server, an upper visual attribute of the added new visual attribute among a plurality of image data for learning included in the training dataset. and transmitting only the image data for training including the code value corresponding to the individual characteristics of .

For example, the added new visual attribute is 'Shoulder', which is a lower visual attribute for the upper individual characteristic of the upper visual attribute (clothing type), and is a specialization applicable only to the 'top' type object. In the case of the visual attribute, from among the plurality of training image data included in the pre-established training dataset, training image data having appearance descriptive data including 'Zb01 to Zb03', which is a code value corresponding to the upper individual characteristic 'top' By extracting and transmitting only this to the expert client in charge of 'shoulder shape', unnecessary data transmission can be minimized.

The server receives new individual characteristics for new visual attributes of each training image data from the expert client (S600), and extracts code values corresponding to a plurality of new individual characteristics received by the server to describe the appearance of each training image data It is added to the data (S700).

In an embodiment, the training dataset may be simply updated by adding only code values corresponding to new individual characteristics to the outline description data of each of the plurality of training image data included in the pre-established training dataset.

For example, the outline descriptive data of specific training image data included in the established training dataset is “Ba01, Bb02, Bg01, Ie01, Ob01, Zb01”, and the training image data is set to a new visual attribute 'shoulder shape ( Shoulder)' is sent to the expert client to receive the new individual characteristic 'plain shoulder', and if the corresponding code value is 'Bd01', the corresponding code value is “Ba01, Bb02, Bd01, Bg01, Ie01, Ob01, Zb01”. By adding only the code value to the existing external description data, it is possible to update the training dataset simply.

In an embodiment, when the image data is moving picture data including a plurality of frames, the step of constructing the training dataset may be constructed by acquiring individual characteristics for each frame in the moving picture data.

Specifically, it is possible to receive a plurality of individual characteristics for each frame in the moving picture data, and to generate the outline description data for each frame sequentially.

Also, in an embodiment, it may be constructed by acquiring dynamic individual characteristics according to dynamic visual attributes for a plurality of frames in the moving picture data.

Specifically, in the case of a surgical image, individual characteristics of visual properties of the object appearance in a stationary state such as 'surgery tool' and 'organ' are acquired for each frame in the video data, but 'movement of the surgical tool' and Similarly, the dynamic individual characteristics of dynamic visual properties for describing the movement of an object are to construct a training dataset by acquiring a plurality of frames in a specific section.

An apparatus for constructing a dataset for learning a visual characteristic recognition model according to another embodiment of the present invention includes one or more computers, and performs the above-described method for constructing a dataset for learning.

The method of constructing a dataset for learning a visual characteristic recognition model of the present invention described above may be implemented as a program (or application) and stored in a medium in order to be executed in combination with a computer, which is hardware.

The above-described program is C, C++, JAVA, machine language, etc. that a processor (CPU) of the computer can read through a device interface of the computer in order for the computer to read the program and execute the methods implemented as a program It may include code (Code) coded in the computer language of Such code may include functional code related to functions defining functions necessary for executing the methods, etc. can do. In addition, the code may further include additional information necessary for the processor of the computer to execute the functions or code related to memory reference for which location (address address) in the internal or external memory of the computer to be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server at a remote location in order to execute the functions, the code is executed by using the communication module of the computer to execute the functions of the computer or server 10 at a remote location. ), etc., may further include a communication-related code for how to communicate, and what information or media to transmit and receive during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, a cache, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program may be stored in various recording media on the various servers 10 accessible by the computer or in various recording media on the computer of the user. In addition, the medium may be distributed in a computer system connected by a network, and a computer readable code may be stored in a distributed manner.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (10)

an image data acquisition step for learning, in which the server acquires a plurality of image data for learning including one or more objects;
Wherein the server transmits the training image data to a plurality of expert clients according to the image type of each training image data, wherein the expert client is a client of an expert who is in charge of classifying each visual attribute in the visual attribute combination, learning image data transmission step;
an individual characteristic receiving step of receiving, by the server, individual characteristics for visual attributes of each image data for learning from the expert client; and
The server generates external description data for each image data for learning based on a plurality of individual characteristics received for each image data for learning, and constructing a data set for learning,
The visual attribute is a specific classification criterion for describing the visual characteristics of the object, and includes a plurality of individual characteristics expressing various visual characteristics within the same classification criterion,
The visual attribute combination is a set of a plurality of visual attributes to be calculated for a specific image type,
The training dataset is used for learning one or more visual characteristic recognition models,
The visual characteristic recognition model is to be learned using a plurality of training image data included in the training dataset and one or more individual characteristics extracted from the appearance description data corresponding thereto, constructing a visual characteristic recognition model training dataset Way. According to claim 1,
The visual characteristic recognition model includes a plurality of individual attribute recognition modules for determining different visual attributes,
The plurality of individual attribute recognition modules are learned based on an individual learning dataset corresponding to each individual attribute recognition module,
The individual learning dataset is a method of constructing a dataset for learning a visual characteristic recognition model in which individual characteristics for a specific visual attribute extracted from external description data are matched to a plurality of training image data. According to claim 1,
The visual attribute combination is a method of constructing a dataset for learning a visual characteristic recognition model, characterized in that it is combined into a plurality of detailed visual attributes that are the lowest visual attributes belonging to each visual attribute. According to claim 1,
The appearance description data is an outline description data in the form of a code string obtained by extracting and combining code values corresponding to a plurality of individual characteristics of the image data for learning, a method of constructing a dataset for learning a visual characteristic recognition model. 5. The method of claim 4,
The code value corresponding to the detailed individual characteristic includes individual characteristic information on one or more upper visual attributes of the detailed visual attribute. According to claim 1,
When a new visual attribute is added,
an additional transmission step of training image data, in which the server transmits, by the server, a plurality of training image data included in the training dataset to a client of an expert who classifies the added new visual attribute in charge;
receiving, by the server, a new individual characteristic for each new visual attribute of each training image data from the expert client; and
Extracting code values corresponding to the plurality of new individual characteristics received by the server and adding them to the external description data of each image data for learning; further comprising, a method of constructing a dataset for learning a visual characteristic recognition model. 7. The method of claim 6,
The method further comprising the step of adding, by the server, the new visual attribute to a visual attribute combination for one or more image types;
The additional transmission step of the image data for training is,
A method for constructing a dataset for learning a visual characteristic recognition model, characterized in that only the image data for learning of the image type to which the new visual attribute is added is transmitted. According to claim 1,
The visual attributes include dynamic visual attributes for describing the movement of a specific object,
When the video data is video data including a plurality of frames,
The step of constructing the training dataset is,
A method for constructing a data set for learning a visual characteristic recognition model, characterized in that it is constructed by acquiring individual characteristics for each frame in the moving picture data and acquiring dynamic individual characteristics for a plurality of frames. A data set construction server device for learning a visual characteristic recognition model, comprising one or more computers, and executing the method of any one of claims 1 to 8. Combined with a computer that is hardware, and stored in a medium to execute the method of any one of claims 1 to 8, a data set construction program for learning a visual characteristic recognition model.

Images