KR102524049B1 - Device and method for recommending apparel for user based on characteristic information - Google Patents

Abstract

The present invention relates to a user coordination recommendation method based on characteristic information of an object. The method includes recognizing, by a server, each of a plurality of objects included in image data; evaluating, by the server, a combination of the plurality of objects based on a deep learning-based object combination evaluation model; and providing an evaluation result to the user, wherein the combination evaluation step calculates characteristic information for each of the plurality of objects, inputs the calculated characteristic information into the object combination evaluation model, and inputs the plurality of object combination evaluation model. to evaluate the combination of

Description

Apparatus and method for recommending user coordination based on object property information

The present invention relates to a method and apparatus for recommending user coordination based on object characteristic information. More specifically, the individual appearance characteristics and abstract characteristics of each object are identified as the object's characteristic information, and a combination of the plurality of object's characteristic information is evaluated to recommend an appropriate object combination to the user.

Recently, with the development of the Internet, social media network services are rapidly growing. As a result, as the amount of multimedia explodes, an effective image retrieval system is required, and the importance of image annotations is gradually increasing due to the need for efficient image retrieval according to the explosively increasing web images.

Most of the image retrieval studies have been carried out in a content-based image retrieval (CBIR) method, which mainly analyzes the content of an image. Content-based image retrieval analyzes the content of an image using visual characteristics such as color, texture, and shape. This method works well when the number of tags defined is small, but performance deteriorates as the dataset grows and the types of tags become more diverse.

Text-based image retrieval (TBIR) is a method of retrieving an image corresponding to text by using text as a query. In this way, the visual content of an image is represented by a manually tagged text descriptor, and is used to perform image retrieval in a dataset management system. That is, in the existing image or video search method, a search is performed based on information directly tagged by a user. In addition, a method of obtaining user interest information through an image also has a problem in that the acquisition is performed based on information directly tagged by the user, and the acquisition result is inaccurate when the user incorrectly tags a keyword in the image. In addition, since there may be differences in keywords defined for each user, there is a problem in that results provided depending on keywords selected by a user inputting an image are different.

Therefore, a method and apparatus for automatically searching an image by extracting the characteristics and attributes of an image without the number of tags or user intervention, or identifying an image based on the corresponding characteristics and attributes are required, but have not yet been proposed. .

On the other hand, various methods of evaluating an existing user's coordination have been proposed. However, in the case of Cody, subjective judgment is inevitably involved, and different results are provided depending on the program or subject being evaluated. That is, it provides inconsistent evaluation results. To solve this problem, it is necessary to accurately extract the characteristic information of the clothes worn by the user and determine the coordination based on the information. However, a method for extracting accurate and consistent characteristic information about the clothing itself has not yet been proposed.

Republic of Korea Patent Publication No. 10-2018-0133200, (2018.12.13)

To solve the above problems, the present invention provides an apparatus and method for recommending user coordination based on characteristic information of an object.

However, the problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A method for recommending user coordination based on object characteristic information according to an aspect of the present invention for solving the above problems includes recognizing, by a server, each of a plurality of objects included in image data; Evaluating a combination between the plurality of objects based on an evaluation model and providing, by the server, the evaluation result to the user, wherein the evaluating the combination includes characteristic information for each of the plurality of objects. and evaluating the combination between the plurality of objects by inputting the calculated characteristic information into the object combination evaluation model.

In one embodiment of the present invention, the characteristic information includes a plurality of individual external appearance characteristics of the object, and the combining evaluation step includes identifying each object type for the plurality of objects included in the image data. inputting the image data into an appearance feature recognition model to calculate individual appearance characteristics for each object for a plurality of appearance classification criteria corresponding to each object type information, and a plurality of individual appearance characteristics of each object and evaluating the combination between the objects by inputting the combined data into the object combination evaluation model.

In one embodiment of the present invention, the characteristic information further includes abstract characteristics of the target object, and the combination evaluation step inputs the individual appearance characteristic information on each object to an abstract characteristic recognition model to determine the abstract characteristics of each object. The method may include calculating abstract characteristics and evaluating combinations between the objects by inputting a plurality of individual external appearance characteristics of each object and data obtained by combining the abstract characteristics into the object combination evaluation model.

In one embodiment of the present invention, in the combination evaluation step, the server calculates an evaluation value for each of the plurality of individual appearance characteristics and the abstract characteristic between the plurality of objects, and the evaluation value is less than a preset reference value. The number of appearance characteristics and abstract characteristics may be calculated, and a combination between the plurality of objects may be evaluated based on the number of appearance characteristics and abstract characteristics.

In one embodiment of the present invention, in the providing step, the server may provide the user with information indicating evaluation grades corresponding to the calculated number among a plurality of predetermined evaluation grades.

In one embodiment of the present invention, the providing step may include selecting a reference object from among the plurality of objects, and searching for a plurality of individual appearance characteristics and abstract characteristics of the recommendation object matching the reference object, and the individual appearance The method may further include providing information on a recommendation target based on characteristics and abstract characteristics to the user.

As an embodiment of the present invention, the server may further include acquiring information on a plurality of objects possessed by the user, and the providing step may include a plurality of individual appearance characteristics of the recommendation object among object information possessed by the user. and at least one object information corresponding to an abstract characteristic may be provided to the user.

In an embodiment of the present invention, the deep learning-based object combination evaluation model may be pre-learned based on image data in which a plurality of objects corresponding to different object types are combined.

A user costume combination evaluation server device based on object characteristics according to another aspect of the present invention for solving the above problems may include one or more computers and execute the above-described user costume combination evaluation method based on object characteristics.

A computer program according to another aspect of the present invention for solving the above problems may be stored in a computer readable recording medium in order to execute a user coordination recommendation method based on object characteristic information.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

According to the present invention as described above, a user's coordination may be evaluated, and a coordination suitable for the user may be recommended based on the evaluation result.

On the other hand, by identifying the individual characteristic recognition of the clothes worn by the user and evaluating the coordinates based on the combination of the individual characteristic recognition information, objectivity is sought in the evaluation result.

In addition, by recognizing the abstract characteristics of the user's coordination based on the recognition of individual characteristics, the suitability of the combination is evaluated more based on the emotion felt from the object, not based on the external features of the object, which is the clothing. Through this, you will be able to evaluate coordination in detail.

In addition, appropriate coordination for each object may be suggested to the user by searching for individual appearance characteristics and abstract characteristics of the matching recommendation object.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary view schematically illustrating a user coordination recommendation system based on characteristic information of an object according to an embodiment of the present invention.
2 is a configuration diagram of an apparatus for recommending user coordination based on property information of an object according to an embodiment of the present invention.
3 is a flowchart schematically illustrating a user coordination recommendation method based on property information of an object according to an embodiment of the present invention.
4 is an exemplary view illustrating separating and recognizing a plurality of objects in image data according to an embodiment of the present invention.
5 is a flowchart schematically illustrating a method of evaluating a combination of objects based on a combination of individual appearance characteristics according to an embodiment of the present invention.
6 is a schematic configuration diagram of a subject combination evaluation model according to an embodiment of the present invention.
7 is a flowchart schematically illustrating a method of evaluating a combination of objects based on a combination of individual appearance characteristics and tracking characteristics according to an embodiment of the present invention.
8 is an exemplary diagram illustrating that scores for emotional characteristics are set according to individual external appearance characteristics according to an embodiment of the present invention.
9 is an exemplary view illustrating that individual external appearance characteristics for each object are calculated by a server and abstract characteristics for the entire object combination are calculated according to an embodiment of the present invention.
10 is an exemplary diagram illustrating providing a user with an object combination evaluation result according to an embodiment of the present invention.
11 is a flowchart schematically illustrating a method of recommending a recommendation object for a reference object according to an embodiment of the present invention.
12 is an exemplary view illustrating recommending a recommendation target for a reference target according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Prior to the description, the meaning of the terms used in this specification will be briefly described. However, it should be noted that the description of terms is intended to help the understanding of the present specification, and is not used in the sense of limiting the technical spirit of the present invention unless explicitly described as limiting the present invention.

In this specification, 'computer' includes all various devices that can perform calculation processing and provide results to users. For example, a computer includes not only a desktop PC and a notebook (Note Book) but also a smart phone, a tablet PC, a cellular phone, a PCS phone (Personal Communication Service phone), synchronous/asynchronous A mobile terminal of IMT-2000 (International Mobile Telecommunication-2000), a Palm Personal Computer (Palm PC), and a Personal Digital Assistant (PDA) may also be applicable. In addition, when a Head Mounted Display (HMD) device includes a computing function, the HMD device may become a computer. In addition, the computer may correspond to the server 10 that receives a request from a client and performs information processing. In this case, the server 10 may be a server computer of a company that provides a service of evaluating a user's coordination and recommending a suitable coordination based on the image data of the user.

In the present specification, 'object' is an object for which detailed description information is to be generated, and refers to an article of a specific classification or category.

In this specification, 'image data' means a 2D or 3D static or dynamic image including a specific object. That is, 'image data' may be static image data of one frame or dynamic image data (ie, video data) of a plurality of consecutive frames. More preferably, it may be a 2D or 3D static or dynamic image of a user wearing a specific object. For example, assuming that the target objects are "hoodie" and "shorts", the image data may be an image of a user wearing "hoodie" and "shorts".

In this specification, a "client" refers to any device including a communication function that users can install and use a program (or application). That is, the client device may include one or more of telecommunication devices such as smart phones, tablets, PDAs, laptops, smart watches, smart cameras, and the like, and remote controllers, but is not limited thereto.

In the present specification, the 'expert client 30' assigns individual appearance characteristics to image data for learning (ie, labeling of image data for learning) or assigns individual appearance characteristics within unlearned appearance classification criteria to input image data A client of an expert performing a role.

In this specification, 'video data for learning' refers to image data used for training of a learning model.

In the present specification, 'appearance classification criteria' refers to classification criteria for appearance expressions necessary for describing the appearance of a specific object or inserting annotations. That is, the 'appearance classification criterion' is a specific classification criterion for describing the external appearance of a specific target object, and includes a plurality of individual external appearance characteristics expressing various external appearance characteristics within the same classification criterion of the target object.

For example, when the object is clothing, the appearance classification criterion is a classification criterion for the outer appearance of the clothing, and may correspond to a pattern, color, fit, length, and the like. That is, if the appearance classification criteria for a specific object increase, the appearance of a specific article belonging to the object may be described in detail.

In the present specification, 'individual appearance characteristics' means various characteristics included in a specific appearance classification criterion. For example, when the appearance classification criterion is color, individual appearance characteristics refer to various individual colors.

In this specification, an 'abstract characteristic' means an abstract characteristic given to a specific object. For example, the 'abstract characteristic' may be an emotional characteristic (eg, in the case of clothing, emotional or fashionable expression such as vintage) for a specific object. Also, for example, 'abstract characteristics' may mean a shape change or motion when the image data is a video.

1 is an exemplary view schematically illustrating a user coordination recommendation system based on characteristic information of an object according to an embodiment of the present invention.

2 is a configuration diagram of an apparatus for recommending user coordination based on property information of an object according to an embodiment of the present invention.

Referring to FIG. 1 , a user coordination recommendation system based on object property information includes a server 10 and a user terminal 20 .

Referring to FIG. 2 , the server includes a communication unit 110, a memory 120, an object combination evaluation unit 130, and a control unit 140.

Referring to FIG. 1 , the server 10 may include a communication unit 110, a memory 120, an object combination evaluation unit 130, and a control unit 140, and each process of the system is performed through each component. do. However, the server 10 may include fewer or more components than those shown in FIG. 1 .

The communication unit 110 may include one or more modules enabling wireless communication between the server 10 and the user terminal 20 . Also, the communication unit 110 may include one or more modules that connect the server 10 to one or more networks.

Meanwhile, the communication unit 110 may transmit and receive various types of information between the server 10 and the user terminal 20 . The communication unit 110 may use various types of communication networks, for example, wireless LANs such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA). A wired communication method such as communication method or Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), FTTH (Fiber To The Home) may be used.

Meanwhile, the steps shown in FIGS. 3, 6, and 10 may be understood to be performed by a device operated by a company that provides a service that evaluates a user's coordination and recommends a suitable coordination, but is not limited thereto. no. Meanwhile, the above-described device may correspond to a server (hereinafter referred to as server) computer.

In the present specification, “clothing” is described as an example for “object” for convenience of description, but the object includes fashion accessories such as necklaces, watches, sneakers, hats, and glasses, which can be clearly understood by those skilled in the art. there is.

Meanwhile, the user terminal 20 refers to a device that generates image data by photographing a user wearing "clothing". More specifically, although not clearly shown in the drawing, the user terminal 20 includes a camera module and a control unit that generate image data for the user, transmits the generated image data to the server, and from the server to the user. Includes a communication unit for receiving information. In addition, it includes an output unit or/and a display unit for displaying the information received from the server. At this time, the display unit, including the multi-touch sensor module layer, may simultaneously perform the role of the input unit.

The user terminal 20 refers to any device including a communication function capable of installing and using a user coordination evaluation program (or application) based on characteristic information of an object, for example. That is, it may include one or more of telecommunication devices such as smart mirrors, smart phones, tablets, PDAs, laptops, smart watches, smart cameras, and remote controllers, but is not limited thereto.

On the other hand, for convenience of description of the present invention, a "user terminal" in this specification describes a "smart mirror" recently used in a store selling clothes as an example.

3 is a flowchart schematically illustrating a user coordination recommendation method based on property information of an object according to an embodiment of the present invention. 4 is an exemplary view illustrating separating and recognizing a plurality of objects in image data according to an embodiment of the present invention.

First, referring to FIG. 3 , the server recognizes each of a plurality of objects included in image data (S410). The server acquires image data about the user from the user terminal through a communication unit. And the server recognizes a plurality of objects in the corresponding image data. For example, if the object corresponds to clothing, the server may separate and recognize the clothing worn by the user as upper and lower garments. Of course, as described above, since the target object includes fashion accessories such as a watch, a hat, and sneakers, when the user is wearing shoes together, the shoes may also be separately recognized.

Meanwhile, after the server recognizes each of a plurality of objects included in the image data (S410), the server evaluates a combination between the plurality of objects based on a deep learning-based object combination evaluation model (S420).

The process of separating and recognizing a plurality of objects in the image data in step S410 may be performed in a series of processes of evaluating combinations between objects through a deep learning-based object combination evaluation model. That is, if image data is input to the deep learning-based object combination evaluation model, a plurality of objects in the image data can be separated and recognized, and then combinations between the objects can be evaluated.

Meanwhile, in the combination evaluation step (S420), characteristic information for each of a plurality of objects is calculated, and a combination between the plurality of objects is evaluated by inputting the calculated characteristic information to the object combination evaluation model.

Each characteristic information about each object, for example, a hoodie (object 1) and slacks (object 2) is calculated, and when the corresponding characteristic information is combined, it is determined whether the user's coordination is appropriate. Even if clothes belong to the same type, the evaluation of the combination is inevitably different depending on the color, pattern, material, etc. of the clothes. Accordingly, the server extracts characteristic information for each object and determines whether a combination of the extracted characteristic information is appropriate.

Referring back to FIG. 3 , the server determines whether the combination of objects is appropriate and then provides the evaluation result to the user (S430). In this case, the server may recommend an appropriate object combination to the user based on the evaluation result. This will be described later in detail.

Meanwhile, hereinafter, a specific method for evaluating a combination of objects will be described.

5 is a flowchart schematically illustrating a method of evaluating a combination of objects based on a combination of individual appearance characteristics according to an embodiment of the present invention. 6 is a schematic configuration diagram of a subject combination evaluation model according to an embodiment of the present invention.

First, the above-described characteristic information of each object includes a plurality of individual external appearance characteristics of the object. In order to evaluate a plurality of object combinations, the server identifies object types for each of the plurality of objects included in the image data (S421).

Specifically, the server identifies each object type of a plurality of objects in the image data of the user obtained from the smart mirror. That is, object type information, which is a specific classification or category of objects included in the image data, is calculated. At this time, according to an embodiment of the present invention, the process of identifying the object type may be performed by inputting image data to a deep learning-based object combination evaluation model. To this end, the deep learning-based object combination evaluation model may further include a module for identifying the object type. For example, referring to FIG. 5 , the server separates and identifies each object worn by the user within image data obtained from the user and simultaneously identifies the type of each object.

Meanwhile, in one embodiment of the present invention, the object type information may be calculated based on a preset object type classification. For example, the upper type classification is set to clothing, accessories, shoes, etc., and the sub-type classification for clothing among the upper types is 'Shirt&Blouse', 'Jacket', 'Coat', 'Pants', 'Skirt', When set to 7 of 'leggings & stockings' and 'one piece', when image data of clothing is input, the object body type recognition model may calculate object type information among the 7 subtypes of clothing.

Referring back to FIG. 5 , the server calculates individual appearance characteristics for each object based on a plurality of appearance classification criteria corresponding to each object type information by inputting the image data to the appearance feature recognition model (S422).

To this end, the deep learning-based object combination evaluation model may further include an individual appearance feature recognition module. The individual appearance feature recognition module includes a plurality of individual feature recognition modules for determining different appearance classification criteria. That is, the object combination evaluation model includes a plurality of individual feature recognition modules specialized in recognizing each appearance classification criterion. As the number of criteria for classifying the appearance of a specific object increases, the server includes a plurality of individual feature recognition modules in the external feature recognition model. The individual feature recognition module calculates individual appearance characteristics included in specific appearance classification criteria of image data.

At this time, each external feature recognition model for each object may be used by sharing a specific individual feature recognition module. For example, when the individual feature recognition module performs color recognition, since the color recognition module can be used universally regardless of the type of object, the server 10 provides a plurality of external feature recognition models differentiated for each object ( 100), a general-purpose color recognition module can be used.

Meanwhile, after calculating the individual appearance characteristics for the appearance classification criteria for each object (S422), data combining a plurality of individual appearance characteristics of each object is input into the object combination evaluation model to evaluate the combination between the objects. (S423).

Specifically, referring to FIG. 4 , the server identifies the types of a plurality of objects in image data as hoodies and training pants, respectively. Then, the individual appearance characteristics for the appearance classification criterion for each hoodie and training pants are calculated. For example, assuming that the individual external characteristics are color, pattern, and material, the individual external characteristics can be calculated as gray, plain, and cotton for hoodies and black, check, and cotton for training pants. In addition, data combining the individual appearance characteristics of the hoodie and the training pants are input into the object combination evaluation model to evaluate the combination between the objects. At this time, information in which the calculated individual appearance characteristics are combined is input to the object combination evaluation model. That is, referring to the above-described example, information combining "hoodie-grey-plain-cotton and training pants-black-check-cotton" will be input to the subject combination evaluation model. To this end, the deep learning-based object combination evaluation model may be a pre-learned model using combination data of individual appearance characteristics of a plurality of objects as one data set. This will be described later in detail.

Meanwhile, as described above, each object may have different appearance classification criteria depending on the object type, and accordingly, individual external appearance characteristics may also be different. For example, in the case of a top, individual appearance characteristics related to the collar and neckline may be calculated, but in the case of a bottom, such individual appearance characteristics cannot be calculated. Accordingly, according to an embodiment of the present invention, the server may evaluate a combination of objects only for the same individual appearance characteristics. That is, the suitability of the combination is evaluated only for individual external appearance characteristics that can be calculated even for different types of objects, such as patterns, materials, and colors.

Meanwhile, as an embodiment of the present invention, the characteristic information may further include abstract characteristics of the target object. That is, the combination of objects is evaluated in consideration of the user's emotional characteristics for each object, rather than evaluating the combination of objects only in terms of external characteristics. Hereinafter, a specific method for evaluating a combination of objects based on abstract characteristics together with individual appearance characteristics will be described.

7 is a flowchart schematically illustrating a method of evaluating a combination of objects based on a combination of individual appearance characteristics and tracking characteristics according to an embodiment of the present invention.

First, referring to FIG. 7 , in order to evaluate a combination of a plurality of objects, the server calculates the abstract characteristics of each object by inputting individual appearance characteristic information about each object into an abstract feature recognition model (S424). . To this end, the deep learning-based object combination evaluation model may further include an abstract feature recognition module.

Meanwhile, in an embodiment of the present invention, the server may further include object type information as well as a plurality of individual appearance characteristics and input the data to the abstract feature recognition model, and outline description data in which individual appearance characteristics or object type information are combined. can be entered in the form of That is, according to the present invention, by calculating abstract characteristics based on appearance description data identified from image data rather than image data itself, it is possible to efficiently process data and calculate objective abstract characteristics.

In one embodiment of the present invention, the abstract characteristic includes a first emotional characteristic. The first emotional characteristic is an emotional characteristic recognized based on the appearance of a specific object, and includes a plurality of first individual emotional characteristics that are specific emotional characteristics.

In one embodiment, the definition and number of each first individual emotional characteristic included in the first emotional characteristic may be set by the server and may be added or changed. Since the first emotional characteristic, which is an emotional characteristic for the external appearance of the object, may be defined differently depending on the era or region, this is to be changed in various ways according to this.

For example, each first individual emotional characteristic may be set to 'cute', 'soft', 'modern', etc. It may be set to further include contrasting first individual emotional characteristics such as 'Elegant', 'Wild', and 'Classic'. However, the first individual emotional characteristic of the present invention is not limited thereto and may be set in various ways.

8 is an exemplary diagram illustrating that scores for emotional characteristics are set according to individual external appearance characteristics according to an embodiment of the present invention.

At this time, the object combination evaluation model sets the score for each of the first individual emotional characteristics set in the plurality of input individual appearance characteristics as the scores for each of the plurality of first individual emotional characteristics are set for each individual appearance characteristic. It may be to calculate the first emotional characteristic by adding them up.

For example, a score for each of the first individual emotional characteristics may be set for each of a plurality of individual appearance characteristics included in each appearance classification criterion.

On the other hand, referring to FIG. 8 , in an embodiment of the present invention, a score table in which a score for each individual appearance characteristic is set for each first individual emotional characteristic is not only one score table, but also a 'V-neck' calculated from specific image data. and 'red', the sum of scores for each first individual emotional characteristic is 'cute: 1', 'elegant: 2', 'soft: 0', 'rough: 1', 'Modern: 0' and 'Classic: 1', and based on this, the first emotional characteristic can be calculated.

In an embodiment, the first emotional characteristic may be calculated by including a ratio of each first individual emotional characteristic score to the total score. For example, in the case of the above example, since the total score is 5, the first emotional characteristic is "cute: 0.2, elegant: 0.4, soft: 0, rough: 0.2, modern" to include the ratio of each first individual emotional characteristic score. Can be calculated as negative: 0, classic: 0.2".

In another embodiment, the first emotional characteristic may be calculated by including each first individual emotional characteristic score. For example, in the case of the above example, the first emotional characteristic is "cute: 1, elegant: 2, soft: 0, rough: 1, modern: 0, classic: 1" to include each first individual emotional characteristic score. " can be calculated.

In another embodiment, the first emotional characteristic may be to calculate only the first individual emotional characteristic for which each first individual emotional characteristic score is equal to or greater than a preset threshold value. For example, in the above example, when the threshold value is 2 (or the ratio is 0.4), only the first individual emotional characteristic of 'elegant' can be calculated as the first emotional characteristic. However, the calculation of the first emotional characteristic is not limited to the above example and may be calculated using various algorithms.

Meanwhile, according to an embodiment of the present invention, the server may calculate abstract characteristics for all combinations of a plurality of objects.

9 is an exemplary view illustrating that individual external appearance characteristics for each object are calculated by a server and abstract characteristics for the entire object combination are calculated according to an embodiment of the present invention.

Referring to FIG. 9 , the server identifies the object type of each object based on image data and calculates individual appearance characteristics of each object. Referring to FIG. 9 , individual external characteristics of a top identified as a shirt or blouse are material, pattern, color, fit, neckline and collar, fastening, and length of the top. At this time, each individual appearance characteristic was calculated as cotton, plain, sky blue, slim fit, boat neckline, pullover, and crop length. On the other hand, the individual external characteristics of the bottoms identified as pants are material, pattern, color, pants fit, and bottom length. And each individual external appearance characteristic was calculated as cotton, plain, gray, baggy pants, and long (maxi). Meanwhile, although not clearly shown in the drawings, abstract characteristics of shirts and pants, which are respective objects, may also be calculated.

Meanwhile, referring to FIG. 9 , all abstract characteristics of the user's coordination are calculated. More specifically, only the abstract characteristics of ACTIVE, CLASSC, ELEGANCE, LOVELY, NATURAL, and SEXY were calculated. At this time, it can be seen that the numerical values for each abstract characteristic are different. This may be calculated as the sum of abstract characteristics calculated for each object. For example, assuming that ACTIVE among the abstract characteristics of the top is calculated as 26 and ACTIVE among the abstract characteristics of the bottom is calculated as 20, the evaluation of the entire combination of top and bottom results in the ACTIVE item being calculated as 46.

Through this, the server may calculate an evaluation result for the user's overall coordination and provide it to the user.

Referring back to FIG. 7 , after calculating the abstract characteristics, the server may evaluate the combination between the objects by inputting data obtained by combining a plurality of individual appearance characteristics of each object and the abstract characteristics into the object combination evaluation model. (S425).

Specifically, it is determined whether the user's coordination is appropriate by inputting data combining abstract characteristics as well as individual appearance characteristics of each object into the object combination evaluation model.

That is, one embodiment of the present invention described above with reference to FIG. 9 identifies the user's coordination and calculates the overall abstract characteristic, whereas the above-described embodiment of the present invention determines whether the user's coordination is appropriate. is to judge Meanwhile, an embodiment of the present invention for determining whether a user's coordination is appropriate by combining individual appearance characteristics and abstract characteristics will be described later in detail.

Meanwhile, as an embodiment of the present invention, the deep learning-based object combination evaluation model may be pre-learned based on image data in which a plurality of objects corresponding to different object types are combined.

Specifically, image data for learning of a deep learning-based object combination evaluation model may be image data in which a plurality of objects belonging to different types are combined. For example, through image data of a combination of upper and lower garments, the object combination evaluation model may extract characteristic information of each upper garment and lower garment and learn whether the combination of the characteristic information is appropriate.

Meanwhile, to this end, the server labels the evaluation result of the object combination based on the individual appearance characteristics and abstract characteristics in the image data for learning through the 'expert client', or the unlearned individual appearance characteristics and Evaluation results can be given for combinations of abstract characteristics. More specifically, the 'expert client' labels evaluation results for each combination of individual appearance characteristics and abstract characteristics of a plurality of objects for each combination. For example, "The individual external characteristics (pattern, color, material) and abstract characteristics of object 1 are check, black, cotton, and chic, and the individual external characteristics (pattern, color, material) and abstract characteristics of object 2 are plain and red. Assuming that it is , woolly, and lovely, the evaluation result of the combination of individual external appearance characteristics and abstract characteristics of the objects 1 and 2 is labeled for each individual external appearance characteristics and abstract characteristics. For example, 1 point is given and 0 points are assigned to characteristics that are judged to be unsuitable, and the total sum is calculated. 1 point will be given for the characteristic, and if it is judged that the combination of black and red is not appropriate for the color of the individual appearance characteristics, 0 points will be given. In this way, scores are calculated for each individual appearance characteristic and abstract characteristic of the object combination. After that, scores for the total feature information are summed, and the summed scores are labeled for the image data as an evaluation result, whereby the deep learning-based object combination evaluation model identifies a plurality of objects in the image data; After extracting the characteristic information (individual appearance characteristics and abstract characteristics) for each object, it is possible to learn to evaluate the appropriateness of the combination of the corresponding characteristic information.

Meanwhile, a method of determining whether the user's coordination is appropriate is as follows.

In one embodiment of the present invention, the server calculates evaluation values for each of the plurality of individual appearance characteristics and the abstract characteristics among the plurality of objects, and the number of individual appearance characteristics and abstract characteristics for which the evaluation values are less than a predetermined reference value may be calculated, and a combination between the plurality of objects may be evaluated based on the number.

As in the learning process of the above-described combination evaluation model based on deep learning, the server calculates evaluation values for each individual appearance characteristic and abstract characteristic among a plurality of objects. For example, in the case of a pattern, '0' is judged to be inappropriate, in the case of a color, '1' is judged to be appropriate, and 'material' is judged to be inappropriate, '1'. If it is judged as '1', each evaluation value is calculated. Then, based on the evaluation values, an overall evaluation value for the object combination is calculated. This may be the sum of each evaluation value. In the case of the above example, the total evaluation value is calculated as 3. On the other hand, in the above description, it has been described that the evaluation value is calculated only with the values of '1' and '0', but it is natural that each evaluation result value can be calculated with various values.

On the other hand, the number of items whose evaluation values are less than a predetermined reference value for each individual appearance characteristic and abstract characteristic is calculated. In the case of the above example, assuming that the preset reference value is 1, the number of items will be calculated as one individual appearance characteristic related to 'material'. A combination between a plurality of objects is evaluated based on the number of objects.

Meanwhile, in one embodiment of the present invention, in the providing step, the server may provide the user with information indicating evaluation grades corresponding to the calculated number among a plurality of predetermined evaluation grades.

10 is an exemplary diagram illustrating providing a user with an object combination evaluation result according to an embodiment of the present invention.

Specifically, as an embodiment of a method of evaluating a combination between a plurality of objects based on the number of individual appearance characteristics and abstract characteristics whose evaluation value is less than a preset reference value, a plurality of evaluation grades are set based on the number. It can be. For example, if the number of items less than a predetermined reference value is 0 to 1, it is set to A grade, 2 to 3 items to B grade, 3 to 4 items to C grade, and 5 or more items to F grade, and the server determines the evaluation result Evaluation grade information corresponding to the number calculated based on is provided to the user.

Referring to FIG. 10, individual appearance characteristics and abstract characteristics, which are characteristic information of a hoodie and training pants worn by a user, are identified, and combination information based on the identification result is input into a deep learning-based object combination evaluation model. It can be seen that one result was output as a C grade. At this time, referring to the above example, it can be inferred that the number of items less than the reference value is 3 to 4 from the evaluation result of the combination of the hoodie and training pants.

As an embodiment of the present invention, the server may provide information about items of individual appearance characteristics and abstract characteristics less than a predetermined reference value to the user based on the above-described evaluation method. For example, if the items that are less than a predetermined reference value, which is the basis for the coordination result of the top and bottom, are "material" and "emotion", the smart mirror is sent to the user that "material" and "emotion" do not match. .

11 is a flowchart schematically illustrating a method of recommending a recommendation object for a reference object according to an embodiment of the present invention. 12 is an exemplary view illustrating recommending a recommendation target for a reference target according to an embodiment of the present invention.

Referring to FIG. 11 , step S430 is a step of selecting a reference object from among the plurality of objects and searching for a plurality of individual external appearance characteristics and abstract characteristics of the recommendation target object matching the reference object (S431) and the individual external appearance characteristics and The method may further include providing information on a recommendation target based on abstract characteristics to the user (S432).

First, in step S431, the server selects a reference object from among a plurality of objects. In one embodiment of the present invention, the reference object may be selected by a user, but is not limited thereto. For example, by accumulating and acquiring image data for each user, the number of times each object is worn is calculated, and based on the number of times that each object is worn, an object that has been worn the most among a plurality of objects is selected as the reference object. It could be.

Meanwhile, after the reference object is selected, the individual appearance characteristics and abstract characteristics of the recommendation object matching the reference object are searched. That is, it is to search for clothing information for proper coordinating. To this end, combination information on the individual appearance characteristics and abstract characteristics of the reference object is input into the deep learning-based object combination evaluation model. Alternatively, although not clearly shown in the drawing, individual appearance characteristics and abstract characteristics of the recommendation object matching the reference object may be searched by inputting the input to an object recommendation model (not shown) included in the object combination evaluation model. .

Meanwhile, it is assumed that the reference object is a hoodie, and that a combination of individual appearance characteristics and abstract characteristics of the corresponding hoodie is identified as "hoodie-gray-cotton-plain-free". At this time, the server identifies a combination of individual appearance characteristics and abstract characteristics of the recommendation target object that matches the reference target object based on the combination information. For example, combination information of individual appearance characteristics and abstract characteristics of a recommendation target object that matches the reference target object may be searched for "training pants-grey-face-check-free".

Meanwhile, after individual appearance characteristics and abstract characteristics are searched (S431), recommendation object information based on the search information is provided to the user (S432). That is, as recommendation object information corresponding to "training pants-gray-cotton-check-free", image data of training pants may be provided to the user. To this end, image data of a plurality of objects may be stored in the memory 120 of the server. At this time, each image data is labeled with information about the individual appearance characteristics and abstract characteristics of the object in the corresponding image data and stored in the memory 120 .

Meanwhile, although not clearly shown in the drawing, as an embodiment of the present invention, the server may further include acquiring information on a plurality of objects owned by the user.

Specifically, the server may obtain information on a plurality of objects owned by the user, that is, clothing, from each user. After receiving each clothing information obtained through the camera module (not shown) of the user terminal 20 through the communication unit, it is matched with each user and stored in the memory 120 . At this time, as an embodiment of the present invention, the above-described information acquisition service of at least one object possessed by the user generates characteristic information on each user's clothing under the name of "personal closet management" and then provides the information to the user. can

Meanwhile, the server may provide the user with at least one object information corresponding to a plurality of individual appearance characteristics and abstract characteristics of the recommendation object among the object information possessed by the user in order to provide the user with the above-described recommendation object information. there is.

Specifically, as described above, when the real-time user's coordinating evaluation based on the user's image data is calculated with a low score, the server searches for a recommendation object that matches the reference object. At this time, based on object information registered for the corresponding user, that is, clothing information, appropriate clothing among clothing currently possessed by the user is extracted as a recommendation object, and the corresponding information is provided to the user. On the other hand, if the user does not have clothing corresponding to the individual appearance characteristics and abstract characteristics of the recommendation object, only the corresponding information may be output and provided to the user.

Meanwhile, in one embodiment of the present invention, the server may provide a linkage service with another user through which each user can share information. For example, under the name of “friend registration”, a service may be provided in which each user links and registers another user information to which object information is to be shared. At this time, a plurality of users registered as friends can share object information registered in the above-described server. The server may search for a recommendation object based on the registered object information of another user registered as a friend, if the user does not have clothes corresponding to the individual appearance characteristics and abstract characteristics corresponding to the recommendation object. . Further, it may be possible to provide the corresponding user with information on holding a recommendation object of another user registered as a friend.

The method for recommending user coordination based on the characteristic information of an object according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: Server
20: smart mirror
110: Communication Department
120: memory
130: object combination evaluation model
131: target body type recognition module
132: Individual appearance characteristic recognition module
133: abstract characteristic recognition module
134: object combination evaluation module
140: control unit

Claims (10)

recognizing, by a server, each of a plurality of objects included in the image data;
evaluating, by the server, a combination between the plurality of objects based on a deep learning-based object combination evaluation model; and
The server providing the evaluation result to a user,
The combination evaluation step,
Calculating characteristic information for each of the plurality of objects, and inputting the calculated characteristic information to the object combination evaluation model to evaluate combinations between the plurality of objects;
The deep learning-based object combination evaluation model is pre-learned based on image data in which a plurality of objects corresponding to different object types are combined,
The characteristic information includes a plurality of individual appearance characteristics and abstract characteristics of the object,
The combination evaluation step,
Calculating object type information for each of the plurality of recognized objects;
Inputting the image data into an appearance feature recognition model to calculate individual appearance characteristics for each of a plurality of recognized objects based on a plurality of appearance classification criteria corresponding to each of the calculated object type information;
Calculating the abstract characteristics of each object by inputting the individual appearance characteristic information on each object to an abstract feature recognition model;
Evaluating the combination between the objects by inputting the combined data of the plurality of individual appearance characteristics and the abstract characteristics of each object into the object combination evaluation model,
When evaluating the combination, the server calculates evaluation values for each of the plurality of individual appearance characteristics and the abstract characteristics among the plurality of objects, and calculates the number of individual appearance characteristics and abstract characteristics for which the evaluation values are less than a predetermined reference value. Calculating and evaluating a combination between the plurality of objects based on the number,
A method for recommending user coordination based on object characteristic information. According to claim 1,
Characterized in that the server calculates the object type information based on a preset object type classification.
A method for recommending user coordination based on object characteristic information. delete delete According to claim 1,
The step of providing,
The server provides the user with information indicating evaluation grades corresponding to the calculated number among a plurality of predetermined evaluation grades.
A method for recommending user coordination based on object characteristic information. According to claim 1,
The step of providing,
selecting a reference object from among the plurality of objects and searching for a plurality of individual appearance characteristics and abstract characteristics of the recommendation object matching the reference object; and
Further comprising the step of providing the user with information on a recommendation target based on the individual appearance characteristics and abstract characteristics.
A method for recommending user coordination based on object characteristic information. According to claim 6,
Further comprising obtaining, by the server, information on a plurality of objects owned by the user,
The step of providing,
Providing the user with at least one object information corresponding to a plurality of individual appearance characteristics and abstract characteristics of the recommendation object among the object information possessed by the user,
A method for recommending user coordination based on object characteristic information. According to claim 1,
The image data for learning of the deep learning-based object combination evaluation model is image data in which a plurality of objects belonging to different types are combined.
A method for recommending user coordination based on object characteristic information. A user costume combination evaluation server device based on characteristic information of an object, including one or more computers, and executing the method of any one of claims 1, 2, and 5 to 8. A computer program stored in a computer readable recording medium to be combined with a computer to execute a user coordination recommendation method based on the characteristic information of an object of any one of claims 1, 2, and 5 to 8.

Images