KR102203355B1 - System and method extracting experience information according to experience of product - Google Patents

Abstract

According to one embodiment of the present invention, a system for extracting experience information according to experience of a product comprises: a storage unit for storing product group experience data corresponding to consumer experience of an experience product group; a text extraction unit for extracting text information of an experiencer from a voice of the experiencer generated according to experience of an experience target product; a feature information extraction unit for performing pretreatment for the text information extracted in the text extraction unit, and extracting feature information that is an input value for machine learning from the pretreated text information; a reaction information extraction unit for extracting experience reaction information that is classified for each category according to experience of the experience target product by applying the extracted feature information to a first model trained based on the product group experience data; and a control unit for matching experiencer identification information of the experiencer and product identification information for the experience target product with the experience reaction information, and controlling the matched experience reaction information to be stored in the storage unit. According to the present invention, it is possible to accurately realize actual perception of consumers about an experience target product.

Description

System and method extracting experience information according to experience of product}

The present invention relates to a technology for providing information related to a product, and more particularly, to a technology for extracting experience information about a product.

In general, products sold to consumers, that is, products or services, are developed and sold to an unspecified number of products in consideration of average consumer preferences or characteristics. However, the characteristics of consumers who consume products vary widely from individual to individual, but since products provided by manufacturing companies or service companies are mass-produced and distributed or provided through a uniform platform, it is difficult to provide products tailored to the characteristics of each consumer. In addition, even if a product reflects the characteristics of a consumer, it is difficult for a manufacturing company or a service company to accurately grasp the characteristics of the consumer for the product.

In particular, in recent years, consumers' desire to experience products is increasing, and experience information based on such experience needs can be used as useful information for companies to develop products. However, it is difficult to collect such consumer experience information, and even if the experience information is collected, fragmented information is only collected manually.

For example, in the case of beauty products (cosmetics), it is desirable for consumers to purchase and use beauty products that match their skin characteristics, but in reality, it is difficult to grasp the characteristics of individual consumers, and due to the distribution system of mass production. Products optimized for the individual characteristics of consumers have not been developed. Recently, beauty product manufacturers have organized experience groups to collect consumer experience information to collect experience information of consumers who use their products, or attempt to collect product experience information through online surveys. Are doing.

However, the information of consumers collected in this way is extremely fragmentary, and thus accurate consumer experience information for the product cannot be provided. Therefore, companies intend to develop products that fit the characteristics of the experiencers, but they are not able to determine the exact direction of development for the products due to the inability to collect consumer experience information.

Korean Patent Application Publication No. 10-2007-0042289 (published on April 23, 2007)

The problem to be solved by the present invention is an experience based on product experience that enables a consumer to extract experience information related to a product by using experience data of a product group to which the experience target product (including tangible product or service product) belongs. It relates to an information extraction system and method.

The experience information extraction system according to product experience according to an embodiment of the present invention for solving the above technical problem includes: a storage unit for storing product group experience data corresponding to the consumer experience of the experience product group; A text extraction unit for extracting text information of the experienced user from the voice of the experienced user generated according to the experience of the product to be experienced; A feature information extracting unit that performs pre-processing on the text information extracted by the text extracting unit, and extracts feature information, which is an input value for machine running, from the pre-processed text information; A reaction information extracting unit for applying the extracted feature information to the first model trained based on the product group experience data, and extracting experience reaction information classified by category according to the experience of the experience target product; And a control unit for controlling the experience-based identification information of the experienced person and product identification information for the experience target product to be matched with the experience response information, and to store the matched experience response information in the storage unit. do.

Further comprising an identification information recognition unit for recognizing the experience person identification information or the product identification information, wherein the control unit initiates an operation of the reaction information extraction unit in response to the recognition of the experience person identification information or the product identification information of the identification information recognition unit. And matching the experience response information extracted by the response information extraction unit with the experience person identification information or the product identification information.

The storage unit may store product group corpus information in which word information related to the experience product group is processed into a corpus for natural language processing and experience information modeling.

The product group corpus information includes the brand or product name of the product to be experienced, the name of the entity representing the brand or product name, the terminology used in the product to be experienced, the term of the component included in the product to be experienced, and the product to be experienced. It is characterized by including at least one or more of functional expression terms, tactile expression terms, and visual expression terms according to their experiences.

The feature information extracting unit may include a preprocessing module for dividing the text information into sentences by using natural language program information for natural language processing, and performing text preprocessing for each of the divided sentences; And a feature vector calculation module that calculates any one of an integer encoding vector, a one-hot encoding vector, and an embedding vector as the feature information for the text pre-processed split sentence. It features.

The feature vector calculation module is characterized in that the embedding vector is calculated by using the product group corpus information in which word information related to the experience product group is processed into a corpus for natural language processing and experience information modeling.

The feature vector calculation module calculates the embedding vector by applying words preprocessed in the preprocessing module to a word embedding matrix of an embedding model modeled by the product group corpus information.

In the first model, after the training data and test data included in the product group experience data are refined by the product group corpus information, the refined training data and the test data are applied according to the experience of the experience target product. It is characterized in that it is modeled to classify the experience response information by category.

The first model includes functional response information according to the experience of the experience target product, tactile response information according to the experience of the experience target product, visual response information according to the experience of the experience target product, and a product different from the experience target product. It is characterized in that the model is modeled to classify by at least one or more reaction information categories among product comparison reaction information according to a value comparison between them and price reaction information for the price of the experience target product.

When the experience target product corresponds to a beauty product, the reaction information extraction unit includes at least one of functional reaction information according to the experience of the beauty product, moisture, application, oil, stickiness, and residual feeling according to the experience of the beauty product. Tactile reaction information including the above, visual reaction information including at least one of brightness, brightness, heaviness, and darkness according to the experience of the beauty product, product comparison based on value comparison between the beauty product and other beauty products It is characterized in that the experiential reaction information is extracted by classifying reaction information and price reaction information for the price of the beauty product according to at least one reaction information category.

Each of the feature vectors of the sentences of the experiential response information extracted by the response information extraction unit for each category is applied to the second model and classified into at least one of a positive emotion category, a negative emotion category, and a neutral emotion category. It characterized in that it further comprises an emotion information extracting unit for extracting the experience emotion information.

The first model or the second model is a recurrent neural network (RNN) model, a long short term memory networks (LSTM) model, a convolution neural network (CNN) model, a gated recurrent unit (GRU) model, and a transformer It is characterized in that any one of the models is used.

The first model or the second model may include a Softmax function, a Hyperbolic Tangent function, a Sigmoid function, a ReLU function, and a function for constructing the deep neural network model. When using any one or more activation functions among the Leaky ReLU functions, but when the sigmoid function or the hyperbolic tangent function is used, a Xavier Initialization method is used for weight initialization, and a relu function Alternatively, when using the Likirelu function, it is characterized in that the He initialization method is used.

The emotion information extracting unit includes the positive emotion category and the negative emotion category for sentences corresponding to functional reaction information, tactile reaction information, visual reaction information, product comparison reaction information, and price response information belonging to the experience reaction information. And extracting the experience emotion information classified into any one of the neutral emotion categories.

It characterized in that it further comprises a sensitivity index calculation unit for calculating a sensitivity index for the experience target product by allocating an emotion score for each of the positive emotion category, the negative emotion category and the neutral emotion category with respect to the experience emotion information. do.

It characterized in that it further comprises a purchase rate calculation unit for calculating a potential purchase rate indicating a potential purchase possibility for the product to be experienced by the experiencer by using the calculated sensitivity index and the actual purchase indices for various products subject to experience by the experiencer. do.

When the experience target product corresponds to a beauty product, the purchase rate calculation unit scales at least one or more of the sensitivity index, the actual purchase index, and the skin suitability index of the beauty product according to the user's skin type and skin tone. A value is applied to calculate the potential purchase rate for the beauty product by the experiencer.

When the experience target product corresponds to a beauty product, skin characteristic information including at least one of skin type information and skin tone information for each body part of the experiencer is measured by measuring the skin for each body part of the experiencer. Further comprising a skin characteristic detecting unit to detect, the control unit, the skin characteristic information and the experience response information corresponding to the experience person identification information and processing the matching processing, and the experience response information and the matching processing to store the skin characteristic information It is characterized in that it is controlled to be stored in the unit.

The storage unit stores beauty attribute information for each beauty product among various experience target products, and uses the beauty attribute information and the skin characteristic information to determine the skin of the beauty product according to the skin type and skin tone of the experiencer. It characterized in that it further comprises a fitness calculator for calculating the fitness index.

The suitability calculation unit may compare the detected skin type information of the experiencer with skin type attribute information of the beauty attribute information of the beauty product, or the skin tone attribute information of the detected skin tone information of the experiencer and the beauty attribute information of the beauty product. It characterized in that to calculate the skin fitness index by comparing.

When the experience target product corresponds to a beauty product, further comprising a recommended product extracting unit for extracting a product recommended to the experiencer by using the sensitivity index and skin characteristic information according to skin measurement for each body part of the experiencer. To do.

It characterized in that it further comprises a user interface unit for outputting one or more of the experience reaction information and the experience emotion information according to the input of any one of the keyword of the experience identification information and the product identification information.

In order to solve the above technical problem, a method for extracting experience information according to a product experience according to another embodiment of the present invention includes: extracting, by a text extraction unit, text information of the experiencer from the experiencer's voice generated according to the experience of the product to be experienced; Performing a pre-processing on the extracted text information by a feature information extraction unit and extracting feature information, which is an input value for machine running, from the pre-processed text information; Applying the extracted feature information to a first model trained on the basis of product group experience data corresponding to the experience product group by a response information extracting unit, and extracting experience response information classified by category according to the experience of the experience target product; And processing, by the control unit, matching the experience response information with the experience response information, and storing the matched experience response information in a storage unit with the experience response information and the product identification information for the experience target product. .

Recognizing the experience person identification information or the product identification information, and extracting the text information after the experience person identification information and the product identification information are recognized.

The extracting of the feature information may include dividing the text information into sentences using natural language program information for natural language processing; Performing text preprocessing for each of the divided sentences; And calculating any one of an integer encoding vector, a one-hot encoding vector, and an embedding vector as the feature information for the text preprocessed segmentation sentence. do.

In the calculating of the feature information, the embedding vector is calculated by using the product group corpus information processed into a corpus for natural language processing and experience information modeling in which word information related to the experience product group is used.

In the calculating of the feature information, the embedding vector is calculated by applying the words preprocessed in the preprocessing module to the word embedding matrix of the embedding model modeled by the product group corpus information.

The product group corpus information includes the brand or product name of the product to be experienced, the name of the entity representing the brand or product name, the terminology used in the product to be experienced, the term of the component included in the product to be experienced, and the product to be experienced. It is characterized by including at least one or more of functional expression terms, tactile expression terms, and visual expression terms according to their experiences.

When the text information belonging to the experience product group includes product group corpus information processed with a corpus for natural language processing and experience information modeling, the first model includes training data and test data included in the product group experience data. After being refined by corpus information, it is modeled to classify the experience response information according to the experience of the experience target product by category by applying the refined training data and the test data.

The first model includes functional response information according to the experience of the experience target product, tactile response information according to the experience of the experience target product, visual response information according to the experience of the experience target product, and a product different from the experience target product. It is characterized in that the model is modeled to classify by at least one or more reaction information categories among product comparison reaction information according to a value comparison between them and price reaction information for the price of the experience target product.

The step of extracting the experience response information includes, when the product to be experienced corresponds to a beauty product, functional reaction information according to the experience of the beauty product, moisture, application, oiliness, stickiness and residual according to the experience of the beauty product. Tactile response information including at least one of the senses, visual response information including at least one of brightness, brightness, heaviness, and darkness according to the experience of the beauty product, value comparison between the beauty product and other beauty products According to the product comparison reaction information and the price reaction information for the price of the beauty product according to at least one reaction information category to extract the experience reaction information by classifying.

The step of applying each of the feature vectors for the sentences of each category of the experience response information to the second model, and extracting the experience emotion information classified into at least one of a positive emotion category, a negative emotion category, and a neutral emotion category. It characterized in that it includes.

The first model or the second model is a recurrent neural network (RNN) model, a long short term memory networks (LSTM) model, a convolution neural network (CNN) model, a gated recurrent unit (GRU) model, and a transformer It is characterized in that any one of the models is used.

The first model or the second model may include a Softmax function, a Hyperbolic Tangent function, a Sigmoid function, a ReLU function, and a function for constructing the deep neural network model. When using any one or more activation functions among the Leaky ReLU functions, but when the sigmoid function or the hyperbolic tangent function is used, a Xavier Initialization method is used for weight initialization, and a relu function Alternatively, when using the Likirelu function, it is characterized in that the He initialization method is used.

The step of extracting the experience emotion information includes the positive emotion category for sentences corresponding to each of functional response information, tactile response information, visual response information, product comparison response information, and price response information belonging to the experience response information, It characterized in that extracting the experience emotional information classified into any one of the negative emotion category and the neutral emotion category.

For the experience emotion information, the step of calculating a sensitivity index of the experience target product by allocating an emotion score for each of the positive emotion category, the negative emotion category, and the neutral emotion category. .

It characterized in that it further comprises the step of calculating a potential purchase rate indicating a potential purchase possibility for the product to be experienced by the experiencer by using the calculated sensitivity index and the actual purchase index for various products to be experienced by the experiencer. .

When the experience target product corresponds to a beauty product, a scaling value is applied to at least one of the actual purchase index, the calculated sensitivity index, and the skin suitability index of the beauty product according to the skin type and skin tone of the experiencer. In this way, the potential purchase rate for the beauty product by the experiencer is calculated.

When the experience target product corresponds to a beauty product, skin characteristic information including at least one of skin type information and skin tone information for each body part of the experiencer is measured by measuring the skin for each body part of the experiencer. Detecting; And matching and processing the skin characteristic information with the experience response information corresponding to the experienced person identification information, and storing the skin characteristic information matched with the experience response information in the storage unit. .

The method further comprising calculating a skin suitability index of the beauty product according to the skin type and skin tone of the experiencer by using the respective beauty attribute information and the skin characteristic information for the beauty products among various experience target products. To do.

The calculating of the skin suitability index may include comparing the detected skin type information of the experienced person with the skin type attribute information of the beauty attribute information of the beauty product, or the detected skin tone information of the experienced person and the beauty attribute of the beauty product. It characterized in that the skin fitness index is calculated by comparing skin tone attribute information among the information.

When the experience target product corresponds to a beauty product, extracting a product recommended to the experiencer using the sensitivity index and skin characteristic information according to skin measurement for each body part of the experiencer .

And outputting at least one of the experience response information and the experience emotion information according to an input of any one of the user identification information and the product identification information.

According to the present invention, by using a model trained to classify experience information about products by category, it is possible to extract the experience information about the experience target product of the experienced person, thereby realizing the actual perception of the consumer about the experience target product. I can figure it out exactly.

In particular, by extracting experience response information, experience emotion information, skin characteristic information, etc. from the experienced person, and extracting information on the sensitivity index, skin fitness index, and potential purchase rate based on the extracted information, it is quantified and effective in relation to the product to be experienced. Consumer experience information can be easily obtained.

In addition, through the quantified consumer experience information, a manufacturer or service company that manufactures or develops an experience target product can set an accurate target customer for a product group related to the experience target product.

1 is a block diagram illustrating a network configuration environment for describing a system for extracting experience information according to a product experience according to the present invention.
FIG. 2 is a block diagram illustrating a system for extracting experience information shown in FIG. 1.
3 is a reference diagram exemplifying product group corpus information for beauty products among the product groups subject to experience.
FIG. 4 is a reference diagram illustrating text information extracted by the text extraction unit shown in FIG. 2.
FIG. 5 is a block diagram showing a connection relationship between detailed components of a feature information extracting unit shown in FIG. 2 and a reaction information extracting unit and an emotion information extracting unit.
6 is a reference diagram schematically illustrating an operation method of a first model implemented as an RNN model.
7 is a reference diagram schematically illustrating an operation method of a first model implemented as an LSTM model.
8 is a reference diagram schematically illustrating an operation method of a first model implemented as a CNN model.
9 is a reference diagram schematically illustrating an operation method of a first model implemented as a GRU model.
10 is a reference diagram schematically illustrating an operation method of a first model implemented as a transformer model.
11 is a flowchart illustrating a method of extracting experience information according to product experience according to the present invention.
12 is a flowchart of an embodiment for explaining the step of extracting feature information shown in FIG. 11.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In addition, the terms "and", "or" include a combination of a plurality of related listed items or any of a plurality of related listed items. In addition, when a component is referred to as being "connected" or "coupled" to another component, it may be directly connected or coupled to the other component, but other components may exist in the middle. It should be understood that there is.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

1 is a block diagram illustrating a network configuration environment for explaining an experience information extraction system (hereinafter referred to as an experience information extraction system) according to a product experience according to the present invention.

Referring to FIG. 1, the network configuration environment includes an experience information extraction system 10, a communication network 20, a portable terminal 30, a product tag 40, and an external terminal 50.

The experience information extraction system 10 extracts product experience information of an experienced person for an experience target product, and provides the extracted product experience information to the external terminal 40. Experience target products may be tangible products (eg, electronic devices, home appliances, cosmetics, beauty products, etc.) or intangible service products (eg, food and beverage services, fitness services, skin care services, etc.). have.

The experience information extraction system 10 can perform short-range wireless communication with the portable terminal 30 and the product tag 40 in order to recognize the portable terminal 30 of the experienced person and the product tag 40 of the product to be experienced. In addition, the portable terminal 30 and the external terminal 50 may be connected to a wired communication network or a wireless communication network to exchange data. A detailed description of the experience information extraction system 10 will be described later.

The communication network 20 includes a wired communication network or a wireless communication network for transmitting and receiving data between the experience information extraction system 10 and the portable terminal 30, the product tag 40, and the external terminal 50. As a wired communication network, it may include an Ethernet communication network or a serial (USB) communication network. The Ethernet communication network is an environment that can be connected using a LAN line, and API can be used by directly connecting to a server on the Internet (HTTP client), and TCP/UDP socket communication can also be performed. The serial (USB) network is asynchronous serial communication, and can be used to connect various sensors to the microcontroller or communicate with other devices. The wireless communication network is a short-range wireless communication network such as Bluetooth and WiFi (wireless fidelity), a portable Internet such as WiBro (wireless broadband internet) or WiMax (world interoperability for microwave access), a global system for mobile communication (GSM), or CDMA. 2G mobile communication network such as (code division multiple access), wideband code division multiple access (WCDMA) or 3G mobile communication network such as CDMA2000, 3.5G mobile such as high speed downlink packet access (HSDPA) or high speed uplink packet access (HSUPA) A mobile communication network for services such as a communication network, a 4G mobile communication network such as a long term evolution (LTE) network or an LTE-Advanced network, and a 5G mobile communication network may be included.

The portable terminal 30 corresponds to a mobile device provided by an experienced person, and may include a smartphone, a tablet PC, or a wearable device. The portable terminal 30 may include a communication module for short-range wireless communication (eg, NFC, Bluetooth, WIFI, etc.). In addition, the portable terminal 30 is device identification information or phone number information of the portable terminal 30 itself or product identification information of the experience target product so that it can be recognized through a reader included in the experience information extraction system 10. For example, barcodes, QR codes, etc.) can be displayed.

The product tag 40 includes a barcode tag, a QR code tag, an RFID tag, an NFC tag, etc. attached to each of the products to be experienced. The product tag 40 may store unique identification information and related information about the product to be experienced. Although one product tag 40 is illustrated in FIG. 1, this is exemplary, and a plurality of product tags may be recognized by the experience information extraction system 10.

The external terminal 50 may correspond to a manufacturer terminal or a service providing terminal that manufactures an experience target product, and may include an online business operator terminal or an offline business operator terminal that sells the experience target product online or offline. The external terminal 50 can make a request for product experience information to the experience information extraction system 10, receive and store product experience information transmitted from the experience information extraction system 10, and use this information to manufacture or sell products. It can be used as information for

FIG. 2 is a block diagram illustrating an embodiment of the experience information extraction system 10 shown in FIG. 1.

2, the experience information extraction system 10 includes a storage unit 100, an identification information recognition unit 102, a text extraction unit 104, a feature information extraction unit 106, and a response information extraction unit 108 , Emotional information extraction unit 110, control unit 112, sensitivity index calculation unit 114, skin characteristics detection unit 116, fitness calculation unit 118, purchase rate calculation unit 120, recommended product extraction unit 122 , A user interface unit 124 and a communication unit 126 may be included.

The storage unit 100 includes information on natural language programs for performing natural language processing, product group experience data corresponding to the consumer experience of the experience product group, or product group corpus information in which word information related to the experience product group is processed for natural language processing and experiential information modeling, etc. Are being saved.

The natural language program information includes various programs for text preprocessing to be described later. For example, the natural language program information may include a sentence tokenization program and a word tokenization program. In addition, natural language program information includes stem extraction program, lemmatiztion program, POS tagging program, stop word removal program, word normalization program, text parsing program, and entity name recognition. It may include a program, an N-Gram program, a BOW program, and the like. In addition, the natural language program information is a package program for processing natural language, and may include NLTK, Spacy, KoNLPy, and the like. In addition, the natural language program information may include Numpy, Pandas, Re, etc. as a package for programming, and may also include Matplotlib, a visualization program, and BeautifulSoup, Selenium, etc. used for web crawling. In addition, various programs for natural language processing may be included in addition to the programs described herein.

The product group experience data includes information according to the experience of the experienced product group (consumer). The product group experience data may include text information, video information, audio information, and the like. In this case, the video information and the audio information finally include the converted text information. The product group experience data may be information including a label or information not including a label according to a modeling method. Product group experience data used in the supervised learning model may include label information corresponding to a correct answer according to learning, and product group experience data used in the unsupervised learning model may include correct answers, that is, data for learning without a label. The product group experience data may be subdivided into index information, text information, label information, and creation time information.

Product group experience data may be collected from various online sites, for example, may be collected by a web crawling method. Here, the sites may be search sites such as Google, Naver, Daum, etc., and may be information collected from community sites of sales sites by product group, online cafe sites by product group, and video sharing sites (for example, YouTube). have.

Product group corpus information corresponds to the corpus of product groups that are commonly used for each experience target product, defines and verifies language rules for product groups, performs grammar correction for product groups, and analyzes statistics such as frequency distribution and simultaneous occurrence of words for product groups. It is a corpus collected to be able to perform.

Product group corpus information is based on the brand or product name of the product to be experienced, the name of the entity representing the brand or product name, the terminology used in the product to be experienced, the term of the component included in the product to be experienced, and the experience of the product to be experienced. Functional expression terms (including functional expression sweet words), tactile expression terms (including tactile expression sweet words), or visual expression terms (including visual expression sweet words) may be included. The product group corpus information may include POS tagging information for a brand or product name, a technical term, an ingredient term, a functional expression term, a tactile expression term, or a visual expression term. In addition, the product group corpus information may include functional expression terms, tactile expression terms, or visual expression terms processed in morpheme units such as free morphemes, derived morphemes, and refractive morphemes.

For example, if the product to be experienced is an electronic device product group (for example, smartphones, tablet PCs, notebook computers, home electronic devices, etc.), the product group corpus information includes brands of various electronic devices, product names of electronic devices, and electronic devices. Technical terms used in the field, functional expression terms according to experiences of electronic devices, tactile expression terms, visual expression terms, and the like may be included. In addition, if the product to be experienced is a household product product group (for example, detergent, toothpaste, shampoo, kitchen utensils, etc.), the product group corpus information is the brand of various household goods, product names of household goods, and specialty used in household goods. It may include terms, functional expression terms, tactile expression terms, or visual expression terms according to experiences of household products. In addition, when the product to be experienced is a group of beauty products (for example, cosmetics, mask packs, beauty devices, etc.), the product group corpus information includes brands of various beauty products, product names of beauty products, and technical terms used in beauty products, It may include functional expression terms according to the experience of beauty products, tactile expression terms (e.g., spreadability, oiliness, moisture, etc.) and visual expression terms (e.g., brightness, dullness, darkness, bling bling, etc.). .

3 is a reference diagram exemplifying product group corpus information for beauty products among the product groups subject to experience. As shown in Fig. 3, as product group corpus information for beauty products, corpus information for brands and product names, corpus information related to comparative investigation (part of speech), corpus information related to terminology, corpus information related to tactile expression terms, It may include corpus information related to visual expression terms, corpus information related to functional expression terms, and the like.

In addition, the storage unit 100 stores attribute information for each experience target product. For example, when an experience target product corresponds to a beauty product, the storage unit 100 stores information such as the composition and ratio of the composition for the beauty product, viscosity, texture type, and content distribution time as beauty attribute information. . In addition, the storage unit 100 is the actual calculated through arithmetic operations on the number of visits of the experiencer visiting the offline store for the experience of various experience target products, the number of purchases of the experiencer, or the time the experiencer stays in the offline store. Save the purchase index. In addition, the storage unit 100 stores various beauty product group information corresponding to skin characteristics for each body part.

In addition, the storage unit 100 may store skin characteristic information for the experienced person, and product experience information including experience response information and experience emotion information may be matched with the experience person identification information and product identification information and stored. Information, skin fitness index information, potential purchase rate information, recommended product information, etc. can be stored.

The identification information recognition unit 102 recognizes the experience person identification information for identifying the experienced person or the product identification information for the product to be experienced, and transmits the recognized experience person identification information or the product identification information to the control unit 110. To this end, the identification information recognition unit 102 may include an experienced person recognition module and a product recognition module.

The experienced person recognition module may include a reader for reading identification information displayed on the portable terminal 30. The reader may read device identification information of the portable terminal 30 or device identification information or phone number information displayed on the portable terminal 30 according to the approach of the portable terminal 30. In addition, the experiencer recognition module may include a biometric recognition module for reading the experiencer's biometric information (eg, the experiencer's face image, the experiencer's fingerprint, etc.).

The product recognition module may include a reader for reading barcodes, QR codes, RFID tags, NFC tags, etc. attached or painted on the product to be experienced. In addition, the product recognition module may read product identification information of an experience target product displayed on the portable terminal 30.

The text extracting unit 104 extracts text information of the experienced user from the voice of the experienced user generated according to the experience of the product to be experienced. To this end, the text extraction unit 104 may include a voice collection module and a text conversion module.

The voice collection module collects voice information of the experiencer from the voice of the experiencer uttered by the experiencer according to the experience of the product to be experienced. The voice collection module performs on and off operations for recording the voice of the experienced user according to the event signal, and collects voice information on the experienced user according to the on and off motion. The voice collection module may recognize an operator's operation of an on button or an off button as an event signal. In addition, the voice collection module may recognize an operation signal for recognizing an experienced person identification information or product identification information of the identification information recognition unit 102 as an event signal for an ON operation of voice recording. According to the event signal for the ON operation, the voice collection module may record a voice made by the experiencer.

The voice collection module may include a voice input device such as a microphone for voice collection. In the recording mode or the voice recognition mode, the voice input unit receives an external sound signal, in particular, an experienced person's voice by a microphone and converts it into an electric voice signal. The voice collection module may include various noise removal algorithms for removing noise generated in a process of collecting voice signals.

The text conversion module converts the collected voice information into text information. The text conversion module converts voice information into text information using a speech to text (STT) engine. Such a text conversion module may convert speech information into text information using a machine learning algorithm. For example, the Cloud Speech-to-Text program provided by Google corresponds to a speech to text (STT) engine using a machine learning algorithm and can be applied to a text conversion module. Text information converted by the text conversion module may include punctuation marks such as periods, exclamation marks, and question marks according to sentences.

4 is a reference diagram illustrating text information extracted by the text extraction unit 104. As shown in FIG. 4, it can be seen that the voice of the experiencer for a beauty product (eg, a foundation product) is collected and converted by the text extraction unit 104 and extracted as text information.

The feature information extracting unit 106 extracts feature information, which is an input value for machine running, from the text information extracted by the text extracting unit 104. To this end, the feature information extraction unit 106 may include a preprocessing module and a feature vector calculation module.

5 is a block diagram illustrating a connection relationship between detailed components of the feature information extracting unit 106 and the reaction information extracting unit 108 and the emotion information extracting unit 110. Referring to FIG. 5, a preprocessing module 106-1 and a feature vector calculation module 106-2 are shown as detailed components of the feature information extracting unit 106.

The preprocessing module 106-1 receives the text information extracted by the text extracting unit 104 (IN), divides it into sentences using natural language program information, and performs text preprocessing for each divided sentence.

The preprocessing module 106-1 may use natural language program information for sentence tokenization in order to divide text information into sentences. The preprocessing module 106-1 is a sentence tokenization program and may divide a sentence based on punctuation marks (eg, hot points) included in text information. In addition, the pre-processing module 106-1 constructs a model for sentence tokenization using machine learning or deep learning technology, in addition to the sentence tokenization method according to the above-described rule-based system, and stores text information in the constructed model. It can be divided into sentences by authorization.

The preprocessing module 106-1 allocates sentence identification information for each divided sentence. The sentence identification information for each divided sentence may be used to identify the divided sentence in the feature vector calculation module 106-2, the response information extracting unit 108, and the emotion information extracting unit 112.

Table 1 below is table information illustrating divided sentences from which the text information shown in FIG. 4 is extracted by the preprocessing module 106-1.

Sentence identification information Split sentence 0000 The Hera Black Cushion can be used without sunscreen. 0001 I have white skin with a little redness, and I usually use No. 17 or higher. 0002 Hera Black Cushion came in various colors. 0003 It's nice that you can choose a variety of colors to suit your skin type. 0004 I tried to buy No. 15 Pink Ivory, but there were many reviews that it was too red, so I chose No. 13 Natural Ivory. 0005 The case is very high-quality, it looks like a luxury cosmetic, but it looks pretty, and the puff isn't stuffy and full, and it's so soft that it's a puff that suits you better with a little rubbing (?) than applying it while tapping it. 0006 13Honde It feels darker than I thought, and it seems to contain a lot of powder. 0007 It feels like something ugly, and it feels like it's powdery on the floor after drinking powdered coffee. 0008 After spending a week at the sea, my hand was burned, so it seems that No. 13 doesn't fit, but the color is pretty (??). 0009 First of all, there is no such thing as a bright feeling or gloss, and it feels a bit like Estee Lauder double wear, so it doesn't feel like natural and inner skin. 0010 It will have good coverage and it will be good in summer when oil is flowing. 0011 I did not apply the foundation properly on the sea, but the floating feeling is strong, so I think it would be good to apply it after firming the foundation. 0012 However, since the price is on the high side, although it is burdensome, it is good compared to overseas brands. 0013 Anyway, there was no darkening and I like the coverage the most.

Referring to Table 1, it can be seen that each sentence identification information is allocated to the divided sentences.

The preprocessing module 106-1 performs a text preprocessing process using product group experience data and/or product group corpus information for each divided sentence.

The preprocessing module 106-1 performs word tokenization using a program for word tokenization for each of the divided sentences. The preprocessing module 106-1 may perform word tokenization based on punctuation marks and spaces. Punctuation marks may include a dot (.), comma (,), question mark (?), semicolon (;), exclamation point (!), and the like. In particular, because Korean is an agglutinating language, unlike English, it includes a survey in words, and spaces are not properly written because the meaning is communicated even without spaces. Accordingly, Korean is a language in which words are not composed of independent words, but is used with investigations, so it is necessary to separate more subdivided words rather than general word units, and the smallest unit of words, morpheme. ) To the unit.

The morphemes are divided into independent morphemes and dependent morphemes. Independent morphemes are morphemes that can be used independently regardless of affixes, endings, and investigations, and become words by themselves, such as body language (noun, pronoun, rhetoric), modifier (tubal, adverb), and interjection. In addition, dependent morphemes are morphemes used in combination with other morphemes. It refers to affix, ending, survey, stem.

Therefore, in the case of Korean, morpheme tokenization, not word tokenization, should be performed for proper word tokenization. To this end, the preprocessing module 106-1 performs morpheme tokenization with reference to product group experience data or product group corpus information for words included in a sentence. For example, the preprocessing module 106-1 is the brand or product name of the product group experience data or the product group experience target products included in the product group corpus information, the entity name representing the brand or product name, and the terminology used in the products subject to experience. , Component terms included in the product to be experienced, functional expression terms (including functional expression sweet words), tactile expression terms (including tactile expression sweet words) or visual expression terms (including visual expression sweet words) Each word included in the sentence is tokenized for each morpheme with reference to the like.

The preprocessing module 106-1 constructs and stores ProductGroupTokenize as a word tokenization program capable of performing morpheme tokenization by applying product group experience data or product group corpus information. ProductGroupTokenize may include an algorithm that performs morpheme tokenization based on a word corresponding to the product group corpus information included in the text information. The preprocessing module 106-1 imports ProductGroupTokenize in order to tokenize text information on a product to be experienced, so that each word can be divided into morphemes from text information.

The preprocessing module 106-1 converts words into normalized words using a word normalization program. Word normalization refers to re-editing words with the same meaning by integrating words with different expressions. The preprocessing module 106-1 may perform stem extraction or lemmatiztion for words in a word normalization method.

The pre-processing module 106-1 extracts stems for tokenized words using a stem extraction (stemming) program. The preprocessing module 106-1 can be extracted using a stem extraction program of NLTK or KoNLPy. Stem refers to a part that, in principle, does not change shape when using a verb (verb, adjective). It corresponds to the part preceding the ending in the application. Stem extraction is a simplified morphological analysis, and it can be said to be a process of cutting out the endings of words by looking at only a set rule. As a stem extraction algorithm, a Porter algorithm, a Lancaster Stemmer algorithm, or the like can be used.

In addition, the preprocessing module 106-1 may extract headwords for words using a lemmatiztion program. Lemma is a basic dictionary-type word. Headword extraction can function to reduce the number of words by finding the root word, even if words have different forms. The preprocessing module 106-1 may use the WordNetLemmatizer algorithm to extract the headword, and in this case, may extract the headword by referring to the part of speech information of the word. The preprocessing module 106-1 extracts the headword in consideration of the context, and accordingly, can preserve the part of speech information of the word.

The preprocessing module 106-1 uses a POS tagging program to tag parts of speech for words. POS tagging is the process of classifying what part of speech was used in the word tokenization process. In this case, the preprocessing module 106-1 may tag parts of speech for words using POS tagging information included in the product group corpus information. The preprocessing module 106-1 can perform POS tagging using the Penn Treebank POS Tags algorithm in NLTK, and POS tagging using algorithms such as Okt (Open Korea Text), Mecab, and Komoran in KoNLPy. have.

The preprocessing module 106-1 may perform a refinement operation of removing stop words using a stop word removal program when unnecessary words are included among words. The preprocessing module 106-1 may be performed prior to the tokenization operation in order to exclude portions that interfere with the tokenization operation and perform the tokenization operation, or may be performed to remove noise remaining after the tokenization operation. have. Noise data to be removed in the refining process may mean letters (special characters, etc.) that are not natural language and have no meaning, but unnecessary words that do not fit the purpose of analysis are removed as noise data. As a method of removing unnecessary words, the preprocessing module 106-1 may remove stopwords, words with a low frequency of appearance, and words with a short length.

The preprocessing module 106-1 may recognize the entity name (Named Entity Recognition) for each word using an entity name recognition program. Entity name recognition is the work of classifying unstructured text into predefined classification systems such as people, groups, places, medical codes, time expressions, quantities, monetary values, and percentages. At this time, the preprocessing module 106-1 may recognize the entity names for words by using entity name information included in the product group corpus information.

Table 2 below is table information illustrating pre-processing results for the divided sentences extracted from Table 1 described above by the pre-processing module 106-1.

Sentence identification information Preprocessing result of split sentences 0000 'Hera','Black Cushion','Sun Cream','Independent','Don't','Right','Become' 0001 'Me','Slightly','Redness','There's','White','Skin','Normal','No. 17','Longer','Write' 0002 'Hera','Black Cushion','Color','Various','Come out' 0003 'Skin','Type','Fit','Color','Various','Choose','There's','Is the same',
'good' 0004 'No. 15','Pink','Ivory','Buy','Has','Too','Red','Late Term','Many','No. 13','Natural','Ivory', 'Select' 0005 'Case','Cham', Luxury','Jida', Highest,'Something','Luxury','Cosmetics','Is like','More','Pretty','Is like','Has', ' 'Puff','good','awesome','feel','no','great','fluffy','hada','tangtang','tap','right','look', 'Slightly','rubbed','more','matched','puff' 0006 'No. 13','Think','See','Dark','Feel','Powder','Many','Contain','Thing','Same' 0007 'Something','Tough','Feel','Powder','Coffee','Ride','Drink','Bottom','Powder','Leave','Feel' 0008 'Sea','One week','There','Hand','Black','Ride','No. 13','Right,'Not','The same','Color','Pretty','Reborn All' 0009 'Once dan','bright','feel','yoongwang','mulgwang','yes','at all','no','slightly','estee loader','double','wear', 'It's right','feel','natural','me','skin','is the same','feel','nothing' 0010 'Covering power','Good','Oil','Zhuang','Flowing','Summer','Good','Same' 0011 'Me','sea','above','basic','rightly','don't','right','float','feel','strong','this','some','basic' ','It's solid','do','right','good','thing','is the same' 0012 'Price','Some','Go out','Side','Pay','Become','Have','Overseas','Brand','Compare','Good','Pan' 0013 'Darkening','No','Covering Power','First','Heart','Listen'

Referring to Table 2, the preprocessing module 106-1 shows a result of performing word tokenization, stem extraction, lemmatiztion, and stopword removal for each of the divided sentences. , In addition, a POS tagging program may be used to tag parts of speech for words or to perform Named Entity Recognition.

The feature vector calculation module 106-2 calculates feature vectors for preprocessed words of divided sentences. To this end, the feature vector calculation module 106-2 may use product group experience data or product group corpus information.

The feature vector may include an integer encoding vector, a one-hot encoding vector, or an embedding vector by word embedding. Integer encoding is an encoding method in which a word set (vocabulary) arranged in order of frequency is created and integers are assigned from the lowest number in order of the highest frequency, and a vector expressed in this way is called an integer encoding vector. One-hot encoding is a vector representation of words in which a value of 1 is assigned to the index of the word to be expressed in the word set, and 0 is assigned to the other indices. vector). Word embedding is a method of expressing a word in the form of a dense vector. The dimension of the vector is expressed as a real value instead of 0 and 1, and the expressed vector is called a dense vector or an embedding vector.

The feature vector calculation module 106-2 calculates an integer encoding vector by assigning a unique index to each of the preprocessed words, or calculates a one-hot encoding vector by assigning "1" to the index position of each of the preprocessed words. I can. For example, as shown in Table 2, the feature vector calculation module 106-2 calculates an integer encoding vector for each of the preprocessed words, or a one-hot encoding vector that assigns "1" as a feature value. Can be calculated.

Also, the feature vector calculation module 106-2 may extract the embedding vector by applying preprocessed words to the word embedding matrix modeled by the embedding model. To this end, the feature vector calculation module 106-2 may model a word embedding matrix corresponding to the product group corpus by applying product group corpus information corresponding to the product group corpus commonly used for each experience target product to the embedding model. The feature vector calculation module 106-2 performs a pre-processing process for the product group corpus information before applying the product group corpus information to the embedding model, and applies the one-hot encoding vector for the pre-processed product group corpus information to the embedding model, and uses the word embedding matrix. Can be modeled. In this case, the feature vector calculation module 106-2 may use a Hierarchical Softmax method or a Negative Sampling method in order to reduce the amount of calculation of the word embedding matrix modeled by the embedding model.

Here, the embedding model may include a Word2Vec model, a FastText model, and a Glove model. The Word2Vec model can learn the meaning of words by using the relationship between words in the sentence of the input corpus and adjacent words. Words with related meanings are likely to appear close to each other on the document, so in the process of repeating learning, the two words can gradually have a vector that is close. The feature vector calculation module 106-2 may use one of a continuous bag of words (CBOW) method and a skip-gram method as a learning method of the Word2Vec model. In addition, the FastText model is an improved algorithm of the Word2Vec model, which expresses an embedding vector as the sum of vectors of subwords (substrings) constituting a word. According to this, the FastText model generates a similar word representation even for words with typos, and makes it possible to generate an appropriate embedding vector considering the morphological similarity of words for new words. In addition, the Glove model makes it easy to measure the similarity between the embedded word vectors by defining the objective function so that the dot product of the two embedded word vectors becomes the log value of the probability of simultaneous appearance in the entire corpus. It is an embedding model that can be effectively reflected.

The word embedding matrix refers to a weight matrix of an embedding model that enables a dense vector corresponding to k-dimension to be output with respect to the input of product group corpus information. As shown in Table 2, the feature vector calculation module 106-2 applies the onehot encoding vector of words preprocessed in the preprocessing module 106-1 to the word embedding matrix modeled by the embedding model, thereby preprocessing words. It is possible to calculate each embedding vector corresponding to. In this case, the k-dimension of the embedding vector calculated by the feature vector calculation module 106-2 may be 100 to 300 dimensions.

In addition, the feature vector calculation module 106-2 may extract embedding vectors for pre-processed words using a pre-training model. Here, the pre-training model is a model that is pre-trained on words using a giant corpus, and in this case, the giant corpus may include product group corpus information. Such a pre-learning model may include, for example, a deep learning model such as Bidirectional Encoder Representations from Transformers (BERT), RoBERTa, DistilBERT, ELMO, and XLNet.

The feature vector calculation module 106-2 applies a large corpus containing product group corpus information to deep learning models such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa, DistilBERT, ELMO, XLNet, etc. to perform pre-learning of words. Perform. Thereafter, the feature vector calculation module 106-2 may extract each embedding vector by applying words preprocessed in the preprocessing module 106-1 to the pre-training model trained by learning.

The feature vector calculated by the feature vector calculation module 106-2 is used as an input value of the reaction information extraction unit 108, and the dimension of the feature vector corresponds to the size of the input value of the reaction information extraction unit 108 do. Accordingly, the feature vector calculation module 106-2 may match the dimension of the feature vector to become a feature vector corresponding to the size of the input value according to the machine learning model implementation of the response information extraction unit 108.

The response information extraction unit 108 applies the calculated feature vector to the first model among the machine learning models, and extracts the experience response information classified by category according to the experience of the experience target product. The extraction result OUT1 of the reaction information extraction unit 108 may be transmitted to the control unit 112 and may also be transmitted to the emotion information extraction unit 110.

The response information extraction unit 108 may include a 　 first model composed of 　 input layer, hidden layer and output layer, and this first model is 　 learned 　 model through forward propagation and back propagation. have. Forward propagation is a process of adjusting weights from the input layer to the output layer and performing calculations, and back propagation is a process of re-adjusting the weights by reflecting errors in the direction of the input layer from the output layer. it means. Here, the input layer may be a layer in which feature vectors are rearranged and input by a reshape function, and the output layer is a layer in which feature vectors input to the input layer are classified and output as experience response information. In addition, the hidden layer may be a layer used as a 　 deep neural network 　 disposed between the input layer and the output layer to form a machine learning or deep learning model.

The first model may be a supervised learning model or an unsupervised learning model. The supervised learning model includes label information corresponding to the correct answer according to the big data collected for learning, and the unsupervised learning model refers to a model that is trained without a label. In order to build the first model as a supervised learning model or an unsupervised learning model, product group experience data may be collected from various online sites. Here, the sites may be search portal sites such as Google, Naver, Daum, etc., may correspond to community sites of sales sites according to product groups, and may also be online cafe sites for each product group, or video sharing sites ( Youtube) may be converted to text information.

The first model may divide the collected product group experience data into training data, verification data, and test data for modeling. The training data is data for training to model the first model, and the test data is data for evaluating the accuracy of modeling of the trained first model. In addition, the verification data is not for evaluating the performance of the model, but for adjusting the performance of the model. A model that has been trained with training data verifies accuracy using the verification data and tunes hyperparameters. Once verification is complete, evaluate the model's performance with the test data. If there is not enough data to divide the verification data and test data, k-fold cross verification can be used.

In the first model, training data and test data belonging to the product group experience data used for modeling are refined by the product group corpus information, and then the refined training data and the test data are applied to the experience of the experience target product. It may be modeled to classify the experience response information according to each category.

The above-described preprocessing module 106-1 may refine training data and test data belonging to product group experience data using product group corpus information. That is, the preprocessing module 106-1 performs morpheme tokenization by applying product group corpus information to the training data and test data for the first model, and stems stem extraction (stemming) or headword extraction (lemmatiztion) by a word normalization program. ), and performs part-of-speech tagging for words using a POS tagging program, removes stopwords when unnecessary words are included among words, and executes an entity name recognition program for each word. You can perform the task of assigning an entity name by using.

The first model includes functional response information according to the experience of the experience target product, tactile response information according to the experience of the experience target product, visual response information according to the experience of the experience target product, and between the experience target product and other products. It may be modeled to classify by at least one or more reaction information categories among product comparison reaction information according to value comparison and price reaction information for the price of the experience target product.

The first model is modeled using the functional expression terminology of the corpus information of the product group in order to classify the experience response information extracted according to the experience of the experience target product as functional response information among the classifications by category, and In order to classify it as reaction information, it is modeled using the tactile expression terminology of the product group corpus information, and it may be modeled using the visual expression terminology of the product group corpus information to classify it as visual response information among categories. In addition, the first model is to classify experiential response information as product comparison response information or price response information among categories, such as terminology, component terms, functional expression terms, tactile expression terms, visual expression terms, etc. It may be modeled using.

For example, in the case where the product to be experienced is an electronic device product group, the first model is the product group corpus information, including brands of various electronic devices, product names of electronic devices, individual names of electronic devices, technical terms used in electronic devices, and electronic devices. It may be modeled using functional expression terms, tactile expression terms, or visual expression terms according to experiences of devices. In addition, when the product to be experienced is a household goods product group, the first model is the product group corpus information, which includes brands of various household goods, product names of household goods, individual names of household goods, terminology used in household goods, and household goods. It may be modeled using functional expression terms, tactile expression terms, or visual expression terms according to experience. In addition, when the product to be experienced is a beauty product group, the first model is the product group corpus information, and the brands of various beauty products, product names of beauty products, individual names of beauty products, terminology used in beauty products, and experience of beauty products It may be modeled using functional expression terms, tactile expression terms (e.g., spreadability, oiliness, moisture, etc.) and visual expression terms (e.g., brightness, dullness, darkness, bling bling, etc.).

The first model can be implemented using a deep neural network. For example, the first model is a model including a deep neural network, a recurrent neural network (RNN) model, a long short term memory networks (LSTM) model, a convolution neural network (CNN) model, a gated recurrent unit (GRU) model, or It may be modeled with a transformer model or the like.

The RNN model is a deep neural network model with a cyclic structure in which the result value from the hidden layer node through the activation function is transferred to the output layer, and the result value is transferred back to the next calculation input of the hidden layer node. 6 is a reference diagram schematically illustrating an operation method of a first model implemented as an RNN model.

The first model implemented as an RNN model is an input layer of the RNN model, and when the feature vector (x) extracted by the feature information extraction unit 106 is input, the experience response information (y) is output through the output layer of the RNN model. When the first model is implemented as an RNN model, if the distance between the relevant information and the point where the information is used is long, the gradient gradually decreases during backpropagation and the learning ability may decrease, so the following LSTM model can be implemented. have.

In order to solve the vanishing gradient problem, the LSTM model has a structure in which cell-states are added to the hidden state of the RNN model. Only important information can be selected and transferred to the next state. Since the cell state acts as a kind of conveyor belt, even if the state elapses for quite a long time, the gradient is relatively well propagated. 7 is a reference diagram schematically illustrating an operation method of a first model implemented as an LSTM model.

The first model implemented as an LSTM model consists of three types of gates: MemoryCell, Input Gate, Forget Gate, and Output Gate with a recursive structure inside each LSTM block. May contain units. Even in the case of the first model implemented as the LSTM model, when the feature vector (X) is input to the input layer of the LSTM model, the memory cell, input gate, forget gate, and output gate Gate) and outputs experience response information.

The CNN model is a model that classifies text using filters. To this end, the CNN DMS may consist of a convolutional layer, a pooling layer, and a fully connected layer. 8 is a reference diagram schematically illustrating an operation method of a first model implemented as a CNN model.

The convolutional layer of the first model implemented as a CNN model is a layer that extracts meaningful feature values from input values, and the pooling layer performs sub-sampling on the feature values, and is fully connected. ) The layer functions as a classifier to output experiential reaction information.

The GRU model is a model that reduces the computation of updates to the hidden state while maintaining a solution to the long-term dependency problem of LSTM. 9 is a reference diagram schematically illustrating an operation method of a first model implemented as a GRU model.

The first model implemented as a GRU model may include only two gates, an update gate and a reset gate, instead of three gates, output, input, and erase gates in the LSTM model. Even in the case of the first model implemented as the GRU model, when a feature vector is input to the input layer of the GRU model, experience reaction information is output through the output layer while passing through the update gate and the reset gate.

The transformer model is a deep neural network model that has an encoder-decoder structure that receives an input sequence from an encoder and outputs an output sequence from a decoder, like seq2seq. 10 is a reference diagram schematically illustrating an operation method of a first model implemented as a transformer model. In this transformer model, there may be m encoders and decoders.

The transformer model adds positional information to the embedding vector of each word and uses it as an input of the model to obtain positional information of the word, which is called positional encoding. The transformer model can calculate the positional encoding information with the embedding vector and use the calculated value as an input value of the encoder. A plurality of encoders and decoders constituting the transformer model may be implemented using attention, respectively. The encoder is composed of Multi-Head Self-Attention and a feed forward neural network, and the decoder is Multi-Head Attention, Masked Multi-Head Self-Attention. And a feed forward neural network.

When the first model is implemented as a transformer model, the hyperparameter can define the following. d _model means the size of input and output determined by the encoder and decoder of the transformer model, and can have values such as 256, 512, 1024, etc., and the size of the embedding vector also has these values. num_layers means layers of an encoder and a decoder in the transformer model, and may be composed of 6 to 12 layers. When using attention in a transformer model, num_heads refers to the number of parallel processing when performing attention in parallel by dividing input information into several pieces and combining the result values into one. d _ff denotes the size of the feed forward neural network, that is, the hidden layer existing inside the transformer model, and 1024, 2048, etc. may be used.

The first model is an activation function used in an RNN model, an LSTM model, a CNN model, a GRU model, or a transformer model.Softmax function, Hyperbolic Tangent function, Sigmoid It may be modeled using a function, a ReLU function, or a Leaky ReLU function.

When there are k classes to be classified, the Softmax function is a function that receives k-dimensional vectors, changes the values of all vector elements to values between 0 and 1, and returns k-dimensional vectors again. . Equation 1 below is an equation representing a softmax function.

In an n-dimensional vector, the k-th element is called a _k , and y _k , which is the probability that the k-th class is the correct answer, can be defined as a softmax function.

The sigmoid function refers to a function that outputs an output value of 0 or 1 for an input value (x)). Equation 2 below is an equation representing the sigmoid function.

When the sigmoid function is multiplied by a very small slope close to zero in the backpropagation process, a vanishing gradient problem may occur in which the slope is not transmitted well at the front end. You can also use the (ReLU) function.

The Hyperbolic Tangent function converts the input value (x) into a value between -1 and 1. The following Equation 3 is an equation representing the hyperbolic tangent function.

Unlike the sigmoid function, the hyperbolic tangent function has a larger variation in the return value, so it tends to have fewer gradient loss problems than the sigmoid function.

The ReLU function outputs 0 when a negative number is input, and returns the input value as it is when a positive number (x) is input. The following Equation 4 is an equation representing a relu function.

Since the relu function does not converge to a specific positive value, it is much more useful than the sigmoid function in deep neural networks, and has excellent computation speed.

The Leaky ReLU function is a function that returns a very small number such as 0.001 instead of 0 when the input value (x) is negative. The following Equation 5 is an equation representing the Likirelu function.

a is a hyperparameter that determines the degree of leaky, and can have a value of 0.01, for example.

When the first model is constructed as a deep learning model, a mean squared error (MSE) function or a cross entropy function may be used as a loss function.

The Mean Squared Error (MSE) function can be used to predict a continuous variable. Equation 6 below is an equation representing the mean square error (MSE) function.

y _k means the output value (predicted value), and t _k means the actual value (correct answer).

The Cross Entropy function is a function that makes the output value of the correct answer become the entire value. The following Equation 7 is an equation representing a cross entropy function.

The base of the logarithm is the natural constant e, t _k represents the k-th label (correct answer) of the one-hot vector, and y _k represents the output value (probability value).

In addition, the first model performs an optimizer, a learning method that minimizes the loss function. In this case, as an optimizer method, batch gradient descent and stochastic gradient descent are performed. , SGD), Mini-Batch Gradient Descent, Momentum, Adagrad, RMSprop, and Adam. In particular, it is preferable to use Adam as the first model.

Batch Gradient Descent is an optimizer method that considers the entire data when calculating the loss, and performs all parameter updates once in a single epoch. The batch gradient descent method takes a long time per epoch because it learns by considering the entire data, but it has the advantage of finding a global minimum. Stochastic Gradient Descent (SGD) is a method of calculating only one randomly selected data rather than the entire data when adjusting the parameter value. The parameter variation is unstable, and sometimes the accuracy may be lower than that of the batch gradient descent method, but the speed is faster than the batch gradient descent method. Mini-Batch Gradient Descent refers to a gradient descent method that adjusts the value of a parameter by calculating only for a set amount. The mini-batch gradient descent method has the advantage of being faster than computing the entire data and more stable than SGD. Momentum is a method of reflecting the slope value of the tangent line one step before the slope of the tangent line calculated in SGD by a certain ratio. When the momentum reaches the local minimum, the value is adjusted to get the effect of escaping from the local minimum. In Adagrad, a different learning rate is applied to each parameter, and parameters with many changes are set to have a small learning rate, and parameters with small changes are set to have a high learning rate. RMSprop is a method that improves these shortcomings by replacing them with other formulas in order to compensate for the shortcomings of Adagrad, where the learning rate is too low in the future when learning continues. Adam is a method that seems to combine both RMS prop and momentum, and it is a method to capture both direction and learning rate.

Also, the first model may be a model to which a dropout method is applied. Dropout is a method of not using parts of a neural network in the learning process. Dropout prevents the artificial neural network from becoming too dependent on a specific neuron or a specific combination during training, and since it does not use neurons with random selection each time, it has the same effect as using different neural networks in an ensemble to prevent overfitting. I can. For example, if the dropout ratio is 0.5, it means that half of the neurons are not used at random for each learning process, but only half of the neurons are used. The first model may use the dropout method only when training a neural network and not use it when predicting.

In addition, the first model is a weight initialization method, and may use an Xavier Initialization method or a He initialization method. Xavier Initialization is divided into two cases when initializing with Uniform Distribution or Normal distribution, and uses the number of neurons in the previous layer and the number of neurons in the next layer. Savior initialization balances the gradient variances of multiple layers to prevent certain layers from getting too much attention or from lagging behind others. He initialization, similar to Savior initialization, is divided into two cases: normal distribution and uniform distribution. However, unlike Savior initialization, He initialization does not reflect the number of neurons in the next layer.

In the first model, when using an S-shaped activation function such as a sigmoid function or a hyperbolic tangent function, a savior initialization method can be used, and when using a relu function or a lichirelu function as an activation function, he initializes. Method can be used.

When the experience target product corresponds to a beauty product, the reaction information extraction unit 108 uses the above-described first model to provide functional reaction information according to the experience of a beauty product, tactile reaction information according to the experience of the beauty product, Experience reaction information classified into visual reaction information, product comparison reaction information according to a value comparison between the beauty product and other beauty products, and the price reaction information on the price of the beauty product may be extracted.

The tactile response information is beauty product experience information including moisture, application, oil, stickiness or residual sensation, and the visual response information is beauty product experience information including brightness, brightness, heaviness, or darkness.

Applicability refers to reaction information on the degree to which the skin is applied evenly without any resistance, and moisture refers to reaction information on the degree of moisture felt on the skin. In addition, the residual sensation refers to reaction information on the degree to which the beauty product remains on the skin, and the oil sensation refers to the reaction information on the oil component felt on the skin. In addition, the product comparison reaction information refers to reaction information according to the tactile and visual value judgment that the experiencer feels for a plurality of experience target beauty products.

For example, the text extracting unit 104 extracts text information on the beauty product as shown in FIG. 4, and the specific information extracting unit 106 calculates feature information corresponding thereto, and the response information extracting unit 108 ), the reaction information extraction unit 108 uses the first model, and the following functional reaction information, tactile reaction information, visual reaction information, product comparison reaction information, or price Experience reaction information such as reaction information can be extracted. That is, the response information extraction unit 108 classifies and extracts each of the divided sentences indexed as sentence identification information 0000, 0003, and 0013 in Table 1 as functional response information, and indexes them with sentence identification information 0001, 0002, 0006, and 0008. Each divided sentence is classified and extracted as visual response information, and each divided sentence indexed with sentence identification information 0007, 0010, and 0011 is classified and extracted as tactile response information, and sentence identification information is divided into 0004, 0005, 0009. Each of the indexed split sentences may be classified and extracted as product comparison response information, and the split sentences indexed with sentence identification information 0012 may be classified and extracted as price response information.

The emotion information extraction unit 110 applies each of the feature vectors for the sentences of each category of the experiential response information extracted by the response information extraction unit 108 to a second model of the machine learning model, Experience emotional information classified into at least one or more of negative emotion categories and neutral emotion categories is extracted. The extraction result OUT2 of the emotion information extraction unit 108 is transmitted to the control unit 112.

Experience-sensitive information corresponding to the positive emotional category means emotional information in which the extracted experiential reaction information expresses the consumer's liking, and the experiential emotional information corresponding to the negative emotional information indicates that the extracted experiential reaction information expresses the consumer's discomfort. It means emotional information, and the experiential emotional information corresponding to the neutral emotional category means emotional information in which the extracted experiential reaction information expresses the non-response of the consumer.

The second model is a category corresponding to the experience response information through application of the refined training data and the test data after the training data and test data included in the product group experience data are refined by NLTK corpus information or KoNLPy corpus information. It may be modeled to classify each sentence according to the positive emotion category, the negative emotion category, and the neutral emotion category.

When training data and test data included in the product group experience data are English data, the corresponding English data may be refined processes such as sentence segmentation, word segmentation, and text pre-processing using NLTK corpus information. In addition, when the training data and test data included in the product group experience data are Korean data, the Korean data may be refined processes such as sentence division, word division, and text pre-processing using KoNLPy corpus information. . Here, the text preprocessing may include stem extraction, headword extraction, part-of-speech tagging, stop word removal, and entity name recognition, as described above.

Training data and test data refined by NLTK corpus information or KoNLPy corpus information are applied to the second model to be modeled, and iteration is performed at a certain batch size, and this is epoch ( epoch), the sentences for each category corresponding to the experience response information may be modeled to be classified into a positive emotion category, a negative emotion category, and a neutral emotion category.

The emotion information extraction unit 110 may include a second model composed of a 　 input layer, a hidden layer and an output layer, and the input layer may be a layer in which feature vectors are rearranged and input by a reshape function, and the output layer is input to the input layer. It is a layer in which feature vectors are classified and output as experience emotional information. In addition, the hidden layer may be a layer used as a 　 deep neural network 　 disposed between the input layer and the output layer to form a machine learning or deep learning model.

For modeling the second model, the collected product group experience data may be divided into training data, verification data, and test data. In the second model, training data and test data belonging to the product group experience data used for modeling are refined by the product group corpus information, and then the refined training data and the test data are applied to the experience of the experience target product. It may be modeled to classify the experience emotional information according to each category.

The second model can be implemented using a deep neural network. For example, the second model is a model including a deep neural network and may be modeled as an RNN model, an LSTM model, a CNN model, a GRU model, or a transformer model.

The second model implemented as the RNN model is an input layer of the RNN model, and when the feature vector extracted by the feature information extraction unit 106 is input, the experience emotional information is output through the output layer of the RNN model. In addition, in the second model implemented as an LSTM model, when a feature vector is input to the input layer of the LSTM model, a memory cell, an input gate, a forget gate, and an output gate are connected. It outputs the emotional information of the experience while passing through it. In addition, when a feature vector is input as an input value of the convolution layer of the second model implemented as a CNN model, sub-sampling is performed through the pooling layer, and the fully connected layer is used. Outputs experience-sensitive information. In addition, in the second model implemented as the GRU model, when a feature vector is input to the input layer, experience emotion information is output through the output layer while passing through an update gate and a reset gate. In addition, the second model implemented as a transformer model passes through the hidden layers of a plurality of encoders and decoders, and finally outputs experience-sensitive information.

The second model uses the Softmax function, Hyperbolic Tangent function, Sigmoid function, ReLU function, or Leaky ReLU function as the activation function of the deep neural network. It may be modeled. In addition, when the second model is constructed as a deep learning model, a mean squared error (MSE) function or a cross entropy function may be used as a loss function.

In addition, the second model is a batch gradient descent method, stochastic gradient descent (SGD), and a mini-batch gradient to perform the optimizer, a learning method that minimizes the loss function. Any one of the methods of descent (Mini-Batch Gradient Descent), Momentum, Adagrad, RMSprop, and Adam can be used.

In addition, the second model may be a model to which a dropout method is applied, and as a weight initialization method, an Xavier Initialization method or a He initialization method may be used. Even in the case of the second model, when an S-shaped activation function such as a sigmoid function or a hyperbolic tangent function is used, the savior initialization method can be used. He initialization method can be used.

Using the second model modeled as described above, the emotional information extraction unit 110 responds to each of functional reaction information, tactile reaction information, visual reaction information, product comparison reaction information, or price reaction information belonging to the experience reaction information. With respect to the sentences, it is possible to extract experiential emotional information classified into any one of a positive emotion category, a negative emotion category, and a neutral emotion category.

That is, when the experiential response information is functional response information, the emotion information extraction unit 110 applies a feature vector corresponding to the sentence of the functional response information to the second model, so that the corresponding functional response information is a positive emotion category and a negative emotion. It is possible to classify whether a category corresponds to a category or a neutral emotion category, and extract the classified experience emotion information. In addition, when the experience response information is tactile response information, the emotion information extraction unit 110 applies a feature vector corresponding to the sentence of the tactile response information to the second model, so that the corresponding tactile response information is a positive emotion category. , Negative emotion category or neutral emotion category, and the classified experience emotion information may be extracted. In addition, when the experiential response information is visual response information, the emotion information extraction unit 110 applies a feature vector corresponding to the sentence of the visual response information to the second model, so that the corresponding visual response information is a positive emotion category and a negative emotion. It is possible to classify whether a category corresponds to a category or a neutral emotion category, and extract the classified experience emotion information. In addition, when the experience response information is product comparison response information, the emotion information extraction unit 110 applies a feature vector corresponding to the sentence of the product comparison response information to the second model, so that the product comparison response information is a positive emotion category. , Negative emotion category or neutral emotion category, and the classified experience emotion information may be extracted. In addition, when the experience response information is price response information, the emotion information extraction unit 110 applies a feature vector corresponding to the sentence of the price response information to the second model, so that the price response information is a positive emotion category and a negative emotion. It is possible to classify whether a category corresponds to a category or a neutral emotion category, and extract the classified experience emotion information.

When the experience target product corresponds to a beauty product, the emotion information extraction unit 110 may extract experience emotion information from experience reaction information according to the experience of the beauty product. The emotion information extraction unit 110 may extract functional reaction information according to the experience of a beauty product. In addition, the emotion information extraction unit 110 can extract the experience emotion information classified into any one of positive, negative, or neutral emotion categories for moisture, application, oil, stickiness, or residual feeling corresponding to tactile reaction information. have. In addition, the emotion information extraction unit 110 may extract the experience emotion information classified into any one of positive, negative, or neutral emotion categories with respect to brightness, brightness, heaviness, or darkness corresponding to the visual response information. In addition, the emotion information extraction unit 110 may extract the experience emotion information classified into any one of positive, negative, or neutral emotion categories for product comparison reaction information or price reaction information.

For example, if the response information extraction unit 108 extracts the experience response information for the text information shown in FIG. 4, the emotion information extraction unit 108 is the sentence identification information 0000, 0003 or 0013 classified as functional response information. By applying the feature vector of each of the human sentences to the second model, experience emotional information obtained by classifying each of the positive emotion category, the negative emotion category, or the neutral emotion category for each of the corresponding sentences may be extracted. In addition, the sentiment information extracting unit 108 applies the feature vector of each sentence of sentence identification information 0001, 0002, 0006 or 0008 classified as visual response information to the second model, so that the sentiment category of each of the sentences is classified. Experienced emotional information can be extracted. In addition, the sentiment information extraction unit 108 applies the feature vector of each sentence of sentence identification information 0007, 0010 or 0011 classified as tactile response information to the second model, so that the sentiment category of each of the sentences is classified. Experience emotional information can be extracted. In addition, the sentiment information extraction unit 108 applies a feature vector of each sentence of sentence identification information 0004, 0005, or 0009 classified as product comparison reaction information to the second model, so that the sentiment category of each of the sentences is classified. Experience emotional information can be extracted. In addition, the sentiment information extraction unit 108 may apply a feature vector of a sentence, which is sentence identification information 0012 classified as price response information, to the second model, and extract experience sentiment information in which the sentiment category of the sentence is classified.

The control unit 112 includes a storage unit 100, an identification information recognition unit 102, a text extraction unit 104, and a feature information extraction unit 106 according to a control command signal, an external command signal, or an event signal by a user input. , Response information extraction unit 108, emotion information extraction unit 110, sensitivity index calculation unit 114, skin characteristic detection unit 116, fitness calculation unit 118, purchase rate calculation unit 120, recommended product extraction unit (122), the operation of the user interface unit 124 or the communication unit 126 is controlled.

To this end, at least a part of the control unit 112 may be implemented in software, firmware, hardware, or a combination of at least two or more of them. The control unit 112 may be implemented by, for example, a processor, and may include a program, routine, sets of instructions, or processes for performing one or more functions.

The controller 112 controls to match the experiential person identification information or product identification information for the experience target product with the experience reaction information or the experience emotion information and store them in the storage unit 100. The experienced person identification information is information that can identify the experienced person, and may be the experience person registration information provided from the experience person, phone number information or device identification information corresponding to the experience person's portable terminal 30, or the experience person's biometric information ( For example, it may be an experiencer's facial image, an experiencer's fingerprint, etc.). In addition, the product identification information may be information read from a barcode, a QR code, an RFID tag, an NFC tag, etc. attached or painted on the product to be experienced.

When the identification information recognition unit 102 recognizes the experienced person identification information or product identification information, the control unit 112 controls the start of the operation of the text extraction unit 104 in response thereto, and the feature information extraction unit 106, reaction information The extraction unit 108 and the emotion information extraction unit 110 are controlled to match and store the experience response information or the experience emotion information with the experience person identification information or product identification information.

For example, after the identification information recognition unit 102 reads device identification information or phone number information, etc. of the portable terminal 30 of the experiencer, reads biometric information of the experiencer, or reads product identification information, When the recognized device identification information, phone number information, biometric information, or product identification information is transmitted to the control unit 112, the control unit 112 responds to this, and an operation start control signal for extracting the text from the text extraction unit 104 Is transferred to the text extraction unit 104. Thereafter, as described above, the text extraction unit 104 extracts text information from the voice information on the experienced person, and based on the extracted text information, the feature information extraction unit 106, the response information extraction unit 108, or When the experience reaction information or the experience emotion information is extracted through the emotion information extraction unit 110, the control unit 112 converts the extracted experience response information or the experience emotion information to device identification information, phone number information, biometric information, or product identification information. Matching is performed, and the matched experience response information or experience emotion information is stored in the storage unit 100. Accordingly, the storage unit 100 stores device identification information, phone number information, biometric information, or experience response information or experience emotion information matched with product identification information. The fact that the experience response information or the experience emotion information is matched and stored with the experience person identification information and the product identification information means that information on various experience target products experienced by the experience person is mapped and stored.

The sensitivity index calculation unit 114 calculates a sensitivity index for the experience target product by allocating an emotion score for each category of the experience emotion information. The emotion index calculation unit 114 allocates an emotion score for each of the positive emotion category, the negative emotion category, and the neutral emotion category for the experience emotion information, and senses a weight according to the frequency of appearance of emotion words belonging to the experience emotion information. By reflecting in the score, a sensitivity index for the experience target product of the experiencer may be calculated.

Table 3 below is a table illustrating the emotional scores allocated for each category of experience response information and experience emotional information. The emotional score as described in Table 3 is exemplary, and can be adjusted as necessary.

Positive emotion Neutrality Negative emotion Functional reaction information 5 2 -5 Tactile reaction information 5 2 -5 Visual reaction information 5 2 -5 Product comparison reaction information 6 3 -6 Price response information 8 3 -8

Meanwhile, the weight applied to the emotional score may be set based on the frequency of occurrence of positive emotional words, negative emotional words, or neutral emotional words. The sensitivity index calculation unit 114 may use a BoW (Bag of Words) algorithm, which is a natural language processing algorithm, in order to calculate the frequency of occurrence of positive emotion words, negative emotion words, or neutral emotion words. The BoW algorithm is an algorithm that quantifies text data by calculating the frequency of occurrence of words without considering the order of words to be applied.

The sensitivity index calculation unit 114 assigns a unique integer index to each text information corresponding to the experience emotion information classified by positive emotion category, negative emotion category, or neutral emotion category using the BoW algorithm, and positions each index. By recording the number of occurrences of the word token in, the frequency of occurrence of the word can be calculated. Thereafter, the sensitivity index calculation unit 108 reflects the weight set higher to the emotional score as the calculated frequency of occurrence of the word is higher, and reflects the weight set lower as the frequency of appearance to the corresponding emotional score. For example, the sensitivity index calculation unit 114 may calculate a sensitivity index for the experience target product of the experiencer by multiplying each emotion score by a weight, and summing and averaging each emotion score multiplied by the weight. .

Table 4 below is a table exemplifying setting values of weights that are preset and stored according to the frequency of word appearance.

Frequency One 2 3 4 5 or more weight One 2 4 6 8

Accordingly, even with the same positive feeling, negative feeling, or neutral feeling, the intensity of positive feeling, negative feeling, or neutral feeling can be distinguished according to the frequency of appearance of the word.

For example, in relation to a specific experience target product experienced by the experiencer, positive experience sensibility information (S1) is extracted for functional response information, and neutral experience sensibility information (S2) is extracted for tactile response information. For visual response information, negative experience emotion information (S3) is extracted, neutral experience emotion information (S4) is extracted for product comparison reaction information, and positive experience emotion information (S5) for price response information. Is assumed to have been extracted. In addition, it is assumed that the value of the appearance frequency calculated by the sensitivity index calculation unit 114 for each experience emotional information using the BoW algorithm is as follows. For example, the number of occurrences of the positive word corresponding to S1 is 2, the number of occurrences of the neutral word corresponding to S2 is 1, the number of occurrences of the negative word corresponding to S3 is 3, and the number of occurrences of the neutral word corresponding to S4. It is assumed that the frequency of appearance is 2 and the frequency of appearance of the positive word corresponding to S5 is 3. According to this, the sensitivity index calculation unit 114 is the sum value according to the multiplication of the emotional score and the weight according to each experience emotional information is (5 * 2) + (2 * 1) + (-5 * 3) + (3) *2) + (8*3) = 27 can be calculated. The sensitivity index calculation unit 114 may calculate a sensitivity index of 27/5 = 5.4 by dividing the calculated sum value by 5 corresponding to the number of categories of the experience emotion information.

When the experience target product corresponds to a beauty product, the skin characteristic detection unit 116 measures the skin condition for each body part of the experiencer, and at least one of skin type information and skin tone information for each body part of the experiencer Detects skin characteristic information including abnormalities.

The skin characteristic detection unit 116 may detect any one of various skin types such as dry, neutral, fatty, and complex as a skin type for the experienced person, and may also detect skin tone information indicating the skin color of the experienced person. .

In addition, the skin characteristic detection unit 116 may detect the experience state information on the beauty product experienced by the experienced person for each body part. The experience status information is information indicating a changed status on the skin surface by the experiencer applying a beauty product to the skin. The skin characteristic detection unit 116 detects experience state color information, experience state moisturizing information, experience state elasticity information, and the like for beauty products applied for each body part.

The skin characteristic detection unit 116 has a built-in camera inside the body in order to precisely photograph the skin for each body part such as face or arm, and a monitor for viewing an image is formed on the front side, and a part of the body can be inserted in the rear. It includes a structure that is there. Here, the camera may include one or more lenses, an image sensor, an image signal processor (ISP), or a flash (eg, an LED or xenon lamp). As for the lens, a lens capable of digital zoom can be used, and the flash can illuminate normal light, polarized light, and ultraviolet rays according to the normal mode, pigment mode, and sebum mode, respectively. In the normal light mode, you can see the condition of the skin, pores, wrinkles, dead skin cells, etc., in the polarization mode, you can see pigmentation, spots, blemishes, etc., and in the ultraviolet mode, you can see sebum, pores, troubles, and acne. . The image sensor processes the image formed from the lens. The image sensor may include a Charge Coupled Device (CCD), CMOS, or the like. The ISP can perform functions such as image brightness adjustment, color interpolation, color correction, and color space conversion on the image signal detected by the image sensor.

In addition, the skin characteristic detection unit 116 is provided with a chin rest and a forehead rest respectively formed to support the chin and forehead when the face of the body is inserted at the rear of the body. And a partition wall in which a lens hole through which the lens portion of the camera passes, a flash hole through which the flash light passes, and several UV lamps formed at the rear of the partition wall, and an intersection formed at regular intervals in the lower part It may include a polarization filter and a parallel polarization filter.

In addition, the skin characteristic detection unit 116 includes a corner meter, a hydrometer, a sebum meter, a skin pH meter, a skin temperature meter, a transdermal moisture loss meter ( Tewanmeter), a melanin measuring instrument (Mexameter), an elasticity measuring instrument (Reviscometer), etc. are included to detect the skin moisturizing state, skin elasticity, and skin pigmentation to which the beauty product is applied. In addition, the skin characteristic detection unit 116 detects the curvature and surface condition of the skin to which the beauty product is applied, and detects the condition of the curved surface of the beauty product applied to the face, and the characteristics of the limbus to which the beauty product is applied. You may. In addition, in the case of wrinkles or pores on the skin of the experiencer, the skin characteristic detection unit 116 may detect the condition of the skin wrinkles using the image analysis program using skin color information collected through the face replica.

When the skin characteristic detection unit 116 detects skin characteristic information for the experienced person and transmits the detection result to the control unit 112, the control unit 112 transmits the skin characteristic information to the experience reaction information or experience emotion information corresponding to the user identification information. Matching process with and control to store the matched skin characteristic information in the storage unit 100, and under the control of the control unit 112, the experience reaction information or the skin characteristic information matched with the experience emotion information is stored in the storage unit 100 ).

When the product to be experienced is a beauty product, the suitability calculation unit 118 calculates a skin suitability index of the beauty product according to the experiencer's skin type and skin tone, using beauty attribute information and skin characteristic information. The beauty attribute information corresponds to product component information, function information, etc. for beauty products and is previously stored in the storage unit 100.

The suitability calculation unit 118 compares the detected skin type information of the experiencer with the skin type attribute information of the beauty attribute information of the beauty product, or the skin tone attribute information of the detected skin tone information of the experiencer and the beauty attribute information of the beauty product. By comparing the skin fitness information is calculated. For example, the fitness calculation unit 118 uses the skin surface error value between the skin surface condition value according to the skin type of the experiencer for the beauty product and the skin surface reference value among the component information of the beauty product to be experienced. For this beauty product, the skin fit index of the experienced person can be calculated. In addition, the suitability calculation unit 118 uses a skin color error value between a skin color state value according to the skin tone of the experiencer for the experience target beauty product and a skin color reference value among component information of the experience target beauty product, The skin fit index can be calculated for the target beauty product. In addition, the fitness calculation unit 118 may calculate an average value of the skin surface error value and the skin color error value for the experience target beauty product as the skin fitness index of the experiencer. At this time, it can be determined that the lower the skin fitness index is, the higher the suitability of the experiencer to the beauty product to be experienced.

The purchase rate calculation unit 120 calculates a potential purchase rate defined as a potential purchase functionality for an experience target product by an experienced person. The purchase rate calculation unit 120 may calculate a potential purchase rate by applying a weight to the response index calculated by the response index calculation unit 114 and the actual purchase index. Here, the actual purchase index is a value calculated through an arithmetic operation according to a combination of any one or more of the number of visits of the experiencer visiting the offline store for the experience of various experience target products, the number of purchases of the experiencer, and the purchase product of the experiencer It may be stored in advance in the storage unit 100.

In addition, when the product to be experienced corresponds to a beauty product, the purchase rate calculation unit 120 provides a response index calculated by the sensitivity index calculation unit 114 and the corresponding beauty product of the experienced person calculated by the fitness level calculation unit 118. A potential purchase rate may be calculated by applying a scaling value to at least one of the skin fitness indices. Here, the scaling value may be a value to which a sensitivity index or a skin fitness index is replaced with an reciprocal number, or a value to which a weight is assigned to a value substituted by an reciprocal number.

In addition, the purchase rate calculation unit 120 applies a scaling value to the actual purchase index stored in the storage unit 100, and calculates the potential purchase rate by arithmetically calculating the scaled actual purchase index and the scaled sensitivity index or skin fitness index. I can. For example, the purchase rate calculation unit 120 multiplies the skin fitness index by a scaling value for substituting the skin fitness index with an inverse number, adds the scaled actual purchase index and the sensitivity index to the multiplied value, and converts the added value into a percentage. By doing so, it is possible to calculate the potential purchase rate for the beauty product experienced by the experienced person. The calculated potential purchase rate can be used as data that can determine the degree of purchase intention of a corresponding experience target product.

When the experience target product corresponds to a beauty product, the recommended product extraction unit 122 extracts a product recommended to the experiencer using the sensitivity index and skin characteristic information according to skin measurement for each body part of the experiencer.

The recommended product extraction unit 122 determines whether the calculated sensitivity index is equal to or greater than a predetermined threshold. When the calculated sensitivity index is greater than or equal to a certain threshold, the recommended product extracting unit 122 includes information on a variety of beauty products corresponding to skin characteristics for each body part stored in the storage unit 100, to which beauty products having a sensitivity index equal to or greater than a certain threshold belong. You can extract beauty product groups by body part. After that, the recommended product extraction unit 122 may extract beauty products matching the skin characteristic information detected by the skin characteristic detection unit 116 from the extracted beauty product group for each body part.

The user interface unit 124 outputs experience response information, experience sensibility information, sensitivity index, skin characteristic information, skin suitability information, or potential purchase rate information, etc. according to input of any one of a keyword among experienced person identification information and product identification information.

The user interface unit 124 may include an input module such as a keypad, a jog shuttle, a touch screen, and an input button in order to receive user identification information or product identification information. The user interface unit 124 may be disposed on one side of the system. In addition, the user interface unit 124 provides experience response information, experience sensitivity information, sensitivity index, skin characteristic information, skin suitability information, or potential purchase rate information corresponding to the operation result of each component under the control of the controller 112. Can be printed. The user interface unit 124 may include a display module or an audio output module to output the operation state of each component in an image or audio format.

In addition, the user interface unit 124 may include a module for displaying image data captured by a camera module or a device management module related to image display for outputting skin characteristic information. For example, the user interface unit 124 determines feature points corresponding to skin aging, including skin wrinkles and spots, based on analysis data including grayscale values, density, area, and linearity of pixels having a density equal to or greater than a reference value. Can be detected. The user interface unit 124 may extract feature points from image data by using a feature point extraction algorithm such as a speeded-up robust features (SURF) algorithm in which a Scale Invariant Feature Transform (SIFT) algorithm is accelerated. In addition, the user interface unit 124 may generate a stretching direction image guiding the stretching direction of the skin according to the distribution of feature points corresponding to skin aging. In addition, the user interface unit 124 converts the image data captured by the camera module into a gray scale image or performs normalization processing according to a predetermined reference value, and removes various noises from the image data. Treatment can be carried out. In addition, the user interface unit 124 is a hue parameter value representing the dominant wavelength of light from the image data, a brightness parameter value representing the intensity or amplitude of light, or An image parameter value including a saturation parameter value indicating the purity or bandwidth of light may be extracted. In addition, the user interface unit 124 may remove noise in image data using an impulse noise removal filter, and noise in the image data from which noise is removed using a sharpening spacial filter. It is possible to reinforce the edge of loss due to removal.

The user interface unit 124 may manage and display a skin image by dividing each part of the face (for example, a forehead part, a neck part, a chin part, a glabellar part, a part under the eyes, a part around the nose and mouth, etc.) . That is, the user interface unit 124 receives skin aging information for each part of the face (eg, skin age, aging) from feature points corresponding to the skin aging position of each part of the face as the skin image application is driven. Degree graph) can be created and stored, and skin aging information for each part of the face can be displayed at the request of the user.

The communication unit 126 transmits the experience information on the experience target product experienced by the experienced person to an external terminal. For example, the communication unit 126 transmits experience response information, experience emotion information, sensitivity index, skin characteristic information, skin suitability information, or potential purchase rate information to the manufacturer terminal of the product to be experienced or the online seller terminal of the product to be experienced. I can. To this end, the communication unit 126 may be connected to a manufacturer terminal or an online seller terminal through a wired communication network or a wireless communication network, and may include communication modules supporting a wired communication protocol or a wireless communication protocol.

11 is a flowchart illustrating a method of extracting experience information according to product experience according to the present invention.

First, the experience information extraction system may recognize the experience person identification information for identifying the experience person or product identification information for the experience target product (step S1000). The experience information extraction system may recognize the information by reading device identification information of the portable terminal or reading device identification information or phone number information displayed on the portable terminal according to the access of the portable terminal. In addition, the experience information extraction system may recognize the information by reading biometric information of the experiencer (eg, a face image of the experiencer, a fingerprint of the experiencer, etc.). In addition, the experience information extraction system can read barcodes, QR codes, RFID tags, NFC tags, etc. attached or painted on the product to be experienced, or read product identification information of the product to be experienced displayed on a portable terminal to recognize the information. I can.

After step S1000, if the experience target product is a beauty product, the experience information extraction system measures the skin for each body part of the experiencer, and obtains skin characteristic information for the skin type information or skin tone information for the experiencer. It can be detected (step S1002).

The experience information extraction system can detect any one of a variety of skin types such as dry, neutral, fat, and complex as a skin type for the experienced person, and can also detect skin tone information indicating the skin color of the experienced person. In addition, the experience information extraction system may detect experience state information about the beauty product experienced by the experienced person for each body part. The experience state information is information indicating a state changed on the skin surface by the experiencer applying a beauty product to the skin, and includes experience state color information, experience state moisturizing information, and experience state elasticity information. In addition, the experience information extraction system detects the curvature and surface condition of the skin on which the beauty product has been applied, and can also detect the condition of the curve of the beauty product applied to the face, and the characteristics of the limbus on which the beauty product has been applied. have. In addition, the experience information extraction system may detect the condition of skin wrinkles using an image analysis program using skin color information collected through a face replica plate in the case of wrinkles or pores on the skin of the experiencer.

The experience information extraction system matches and processes skin characteristic information with experience response information corresponding to the experience person identification information, and stores the skin characteristic information matched with the experience response information.

After step S1002, the experience information extraction system calculates the skin suitability index of the beauty product according to the skin type and skin tone of the experiencer, using the beauty attribute information and the skin characteristic information for each beauty product among various experience target products. It can be calculated (step S1004).

The experience information extraction system compares the detected skin type information of the experiencer with the skin type attribute information of the beauty attribute information of the beauty product, or the skin tone attribute information of the detected skin tone information of the experiencer and the beauty attribute information of the beauty product. By comparing the skin fitness index can be calculated.

The experience information extraction system uses the skin surface condition value according to the experiencer's skin type for the beauty product and the skin surface error value between the skin surface reference value among the component information of the beauty product to be experienced. The index of fitness for the skin of the experienced person can be calculated. In addition, the experience information extraction system uses a skin color error value between a skin color condition value according to the skin tone of the experiencer for the experience target beauty product and a skin color reference value among the component information of the experience target beauty product, The skin fit index can be calculated for the beauty product. In addition, the experience information extraction system can calculate the average value of the skin surface error value and the skin color error value for the beauty product to be experienced as an index of skin fitness of the experiencer.

The experience information extraction system matches the calculated skin fitness index with the experience response information corresponding to the experienced person identification information, and stores the skin fitness index matched with the experience response information.

After step S1004, the experience information extraction system may extract text information of the experiencer from the experiencer's voice generated according to the experience of the experience target product (step S1006).

The experience information extraction system performs on and off operations for recording the voice of the experienced user according to the event signal, and collects voice information on the experience user according to the on and off operations. The experience information extraction system may recognize a motion signal for recognizing an experienced person identification information or product identification information as an event signal for an on-motion of voice recording. The experience information extraction system can collect clear speech information through various noise removal algorithms to remove noise generated in the process of collecting speech signals.

The experience information extraction system converts the collected voice information into text information using the STT engine. The experience information extraction system may convert voice information into text information using a machine learning algorithm.

After step S1006, the experience information extraction system may perform pre-processing on the extracted text information, and extract feature information, which is an input value of a machine running model, from the pre-processed text information (step S1008).

FIG. 12 is a flowchart of an embodiment for explaining the step of extracting feature information shown in FIG. 11 (S1008 ).

First, the experience information extraction system may divide the text information into sentences by using natural language program information for natural language processing (step S1100).

The experiential information extraction system can use natural language program information for sentence tokenization in order to divide text information into sentences. The experiential information extraction system is a sentence tokenization program and can divide sentences based on punctuation marks (eg, hot points) included in text information. In addition, the experiential information extraction system constructs a model for sentence tokenization using machine learning or deep learning technology, in addition to the sentence tokenization method according to the above-described rule-based system, and applies text information to the constructed model. It can also be divided into units. The experience information extraction system allocates sentence identification information for each divided sentence. Sentence identification information for each divided sentence may be used to identify the divided sentence.

After step S1100, the experience information extraction system may perform text pre-processing for each of the divided sentences (step S1102).

The experience information extraction system performs a text pre-processing process for each divided sentence using product group experience data and/or product group corpus information. The experiential information extraction system performs word tokenization using a program for word tokenization for each of the divided sentences. The experiential information extraction system can perform word tokenization based on punctuation marks and spaces.

The experience information extraction system may perform morpheme tokenization for words included in a sentence by referring to product group experience data or product group corpus information. For example, the experience information extraction system is the brand or product name of the product group experience data or the product group experience target products included in the product group corpus information, the entity name representing the brand or product name, the terminology used in the products subject to experience, and the experience target. Refer to component terms included in products, functional expression terms (including functional expression sweet words), tactile expression terms (including tactile expression sweet words) or visual expression terms (including visual expression sweet words) according to the experiences of the products to be experienced. Thus, each word included in the sentence can be tokenized for each morpheme.

The experiential information extraction system converts words into normalized words using a word normalization program. The experiential information extraction system can extract stems or headwords for words in a word normalization method.

The experiential information extraction system uses a stem extraction program to extract stems for tokenized words. The experience information extraction system can be extracted using the stem extraction program of NLTK or KoNLPy. The experiential information extraction system can use a porter algorithm, a Lancaster stammer algorithm, and the like as a stem extraction algorithm.

In addition, the experience information extraction system may extract headwords for words using a headword extraction program. The experiential information extraction system can use the WordNetLemmatizer algorithm to extract the headword, and at this time, the headword can be extracted by referring to the part of speech information of the word. The experience information extraction system extracts the headword in consideration of the context, and accordingly, can preserve the part-of-speech information of the word.

The experience information extraction system uses a POS tagging program to tag parts of speech for words. In this case, the experience information extraction system may tag parts of speech for words by using POS tagging information included in the product group corpus information. The experience information extraction system can perform part-of-speech tagging using the Penn Treebank POS Tags algorithm in NLTK, and part-of-speech tagging using algorithms such as Okt, Mecab, and Komoran in KoNLPy.

The experiential information extraction system may perform a refinement operation of removing stop words using a stop word removal program when unnecessary words are included among words. The experiential information extraction system may be performed prior to the tokenization operation in order to exclude parts that interfere with the tokenization operation and perform the tokenization operation, or may be performed to remove the noise remaining after the tokenization operation.

The experiential information extraction system can recognize the entity name for each word using the entity name recognition program. At this time, the experience information extraction system may recognize entity names for words using entity name information included in the product group corpus information.

After step S1102, the experience information extraction system may calculate any one of an integer encoding vector, a one-hot encoding vector, and an embedding vector from the text pre-processed split sentence (step S1104).

The experiential information extraction system can perform integer encoding by assigning a unique index to each of the preprocessed words, or can calculate a one-hot encoding vector by performing one-hot encoding that assigns "1" to the index position of the preprocessed words. .

In addition, the experience information extraction system may extract embedding vectors for pre-processed words from the word embedding matrix modeled by the embedding model. To this end, the experience information extraction system may model a word embedding matrix corresponding to the product group corpus by applying the product group corpus information corresponding to the product group corpus commonly used for each experience target product to the embedding model. The product group corpus information includes the brand or product name of the experience target products, the entity name representing the brand or the product name, the terminology used in the experience target products, the component terms included in the experience target products, and the experience target It may include functional expression terms, tactile expression terms, and visual expression terms according to product experiences.

At this time, the experiential information extraction system performs a pre-processing process on the product group corpus information before applying the product group corpus information to the embedding model, and applies the one-hot encoding vector for the pre-processed product group corpus information to the embedding model to model the word embedding matrix. I can. Here, the embedding model may include a Word2Vec model, a FastText model, and a Glove model.

The word embedding matrix refers to a weight matrix of an embedding model that enables a dense vector corresponding to k-dimension to be output with respect to the input of product group corpus information. The experience information extraction system may calculate each embedding vector corresponding to the preprocessed words by applying a one-hot encoding vector of the preprocessed words to the word embedding matrix modeled by the embedding model.

In addition, the experiential information extraction system may extract embedding vectors for preprocessed words using a pre-learning model. Here, the pre-learning model is a model that has been pre-trained on words using a giant corpus, and at this time, the giant corpus may include product group corpus information. Such a pre-learning model may include, for example, a deep learning model such as BERT, RoBERTa, DistilBERT, ELMO, or XLNet. The experiential information extraction system performs pre-learning on words by applying a huge corpus containing product group corpus information to deep learning models such as BERT, RoBERTa, DistilBERT, ELMO, and XLNet. Thereafter, the experience information extraction system may extract each embedding vector by applying preprocessed words to the pre-learning model trained by learning.

The experience information extraction system may match the dimension of the feature vector so that the extracted feature vector can be used as an input value of a machine learning model for extracting experience response information or experience emotion information.

Meanwhile, after step S1008, the experience information extraction system applies the extracted feature information to the first model trained based on product group experience data corresponding to the experience product group, and experiences classified by category according to the experience of the experience target product. The reaction information can be extracted (S1010 step).

The experience information extraction system may include a first model composed of an input layer, a hidden layer, and an output layer. The input layer may be a layer in which feature vectors are rearranged and input by a reshape function, and the output layer is a layer in which feature vectors input to the input layer are classified and output as experience response information. In addition, the hidden layer may be a layer used as a 　 deep neural network 　 disposed between the input layer and the output layer to form a machine learning or deep learning model.

Product group experience data may be collected from various online sites in order to establish the first model as a supervised learning model or an unsupervised learning model. The first model may divide the collected product group experience data into training data, verification data, and test data for modeling. In the first model, training data and test data belonging to the product group experience data used for modeling are refined by the product group corpus information, and then the refined training data and the test data are applied to the experience of the experience target product. It may be modeled to classify the experience response information according to each category.

The experience information extraction system can refine training data and test data belonging to product group experience data using product group corpus information. That is, the experiential information extraction system applies product group corpus information to the training data and test data for the first model to perform morpheme tokenization, and performs stem extraction or headword extraction by a word normalization program, and a part-of-speech tagging program. Part of speech tagging is performed on words by using this method, and when unnecessary words are included in the words, stop words are removed, and an entity name can be assigned to each word using an entity name recognition program. have.

The first model includes functional response information according to the experience of the experience target product, tactile response information according to the experience of the experience target product, visual response information according to the experience of the experience target product, and between the experience target product and other products. It may be modeled to classify by at least one or more reaction information categories among product comparison reaction information according to value comparison and price reaction information for the price of the experience target product.

The first model is modeled using the functional expression terminology of the corpus information of the product group in order to classify the experience response information extracted according to the experience of the experience target product as functional response information among the classifications by category, and In order to classify it as reaction information, it is modeled using the tactile expression terminology of the product group corpus information, and it may be modeled using the visual expression terminology of the product group corpus information to classify it as visual response information among categories. In addition, the first model is to classify experiential response information as product comparison response information or price response information among categories, such as terminology, component terms, functional expression terms, tactile expression terms, visual expression terms, etc. It may be modeled using.

For example, if the product to be experienced is a beauty product group, the first model is the product group corpus information, and the brands of various beauty products, product names of beauty products, individual names of beauty products, terminology used in beauty products, beauty products It may have been modeled using functional expression terms, tactile expression terms (e.g. spreadability, oiliness, moisture, etc.) and visual expression terms (e.g., brightness, dullness, darkness, bling bling, etc.) according to their experiences. have.

The first model can be implemented using a deep neural network. For example, the first model is a model including a deep neural network and may be modeled as an RNN model, an LSTM model, a CNN model, a GRU model, or a transformer model.

The first model is an activation function used in an RNN model, LSTM model, CNN model, GRU model, or transformer model, which is modeled using a softmax function, a hyperbolic tangent function, a sigmoid function, a relu function, or a lichirelu function. Can be.

When the first model is constructed as a deep learning model, a mean square error (MSE) function or a cross entropy function may be used as a loss function.

In addition, the first model performs an optimizer, which is a learning method that minimizes the loss function. In this case, as the optimizer method, the batch gradient descent method, the stochastic gradient descent method (SGD), the mini-batch gradient descent method, the momentum, Any one of Adagrad, RMS prop, or Adam can be used. In particular, it may be desirable to use Adam as the first model.

Also, the first model may be a model to which a dropout method is applied. The first model may use the dropout method only when training a neural network and not use it when predicting.

In addition, the first model is a weight initialization method, and may use a savior initialization method or a He initialization method. At this time, the first model can use a savior initialization method when using an S-shaped activation function such as a sigmoid function or a hyperbolic tangent function, and when using a relu function or a requirelu function as an activation function, He initialization method can be used.

The experience information extraction system uses the above-described first model when the product to be experienced corresponds to a beauty product, functional reaction information according to the experience of the beauty product, tactile reaction information according to the experience of the beauty product, and visual reaction. Experience reaction information classified into information, product comparison reaction information according to a value comparison between the beauty product and other beauty products, and the price response information on the price of the beauty product may be extracted. The tactile response information is beauty product experience information including moisture, application, oil, stickiness or residual sensation, and the visual response information is beauty product experience information including brightness, brightness, heaviness, or darkness. In addition, the product comparison reaction information refers to reaction information according to the tactile and visual value judgment that the experiencer feels for a plurality of experience target beauty products.

The experiential information extraction system matches the extracted experiential response information with the experiential identification information of the experiential person or product identification information for an experiential product, and stores the matching experiential response information.

After step S1010, the experience information extraction system applies each of the feature vectors for sentences for each category of the experience response information to the second model, and is classified into at least one of a positive emotion category, a negative emotion category, and a neutral emotion category. Experience emotional information can be extracted (step S1012).

Here, the second model corresponds to the experience response information through application of the refined training data and test data after the training data and test data included in the product group experience data are refined by NLTK Corpus information or KoNLPy Corpus information. It may be modeled to classify sentences for each category into the positive emotion category, the negative emotion category, and the neutral emotion category.

When training data and test data included in the product group experience data are English data, the corresponding English data may be refined processes such as sentence segmentation, word segmentation, and text pre-processing using NLTK corpus information. In addition, when the training data and test data included in the product group experience data are Korean data, the Korean data may be refined processes such as sentence division, word division, and text pre-processing using KoNLPy corpus information. . Here, the text preprocessing may include stem extraction, headword extraction, part-of-speech tagging, stop word removal, and entity name recognition, as described above.

The experience information extraction system may include a second model composed of an input layer, a hidden layer, and an output layer, and the input layer may be a layer in which feature vectors are rearranged by a reshape function, and the output layer is a feature vector input to the input layer. It is a layer that is classified and output as experience-sensitive information. In addition, the hidden layer may be a layer used as a 　 deep neural network 　 disposed between the input layer and the output layer to form a machine learning or deep learning model.

For modeling the second model, the collected product group experience data may be divided into training data, verification data, and test data. In the second model, training data and test data belonging to the product group experience data used for modeling are refined by the product group corpus information, and then the refined training data and the test data are applied to the experience of the experience target product. It may be modeled to classify the experience emotional information according to each category.

The second model can be implemented using a deep neural network. For example, the second model is a model including a deep neural network and may be modeled as an RNN model, an LSTM model, a CNN model, a GRU model, or a transformer model.

The second model may be modeled using a softmax function, a hyperbolic tangent function, a sigmoid function, a relu function, or a lichirelu function as an activation function of the deep neural network. In addition, when the second model is constructed as a deep learning model, a mean square error (MSE) function or a cross entropy function may be used as a loss function.

In addition, the second model is a batch gradient descent method, a stochastic gradient descent method (SGD), a mini-batch gradient descent method, a momentum, Adagrad, RMS prop, in order to perform the optimizer, a learning method that minimizes the loss function. Or you can use either of Adam's methods.

In addition, the second model may be a model to which a dropout method is applied, and as a weight initialization method, a savior initialization method or a He initialization method may be used. Even in the case of the second model, when an S-shaped activation function such as a sigmoid function or a hyperbolic tangent function is used, the savior initialization method can be used. He initialization method can be used.

Using the second model, the experience information extraction system uses a positive emotion category for sentences corresponding to functional reaction information, tactile reaction information, visual reaction information, product comparison reaction information, or price reaction information belonging to the experience reaction information. Experience sentiment information classified into either negative emotion category or neutral emotion category may be extracted.

For example, when an experience target product corresponds to a beauty product, the experience information extraction system may extract experience emotion information from experience reaction information according to the experience of the beauty product. The experiential information extraction system can extract experiential emotional information classified into any one of positive, negative, or neutral emotional categories for moisture, spreadability, oiliness, stickiness, or residual feeling corresponding to tactile response information. In addition, the experience information extraction system may extract the experience emotional information classified into one of positive, negative, or neutral emotional categories for light, bright, heavy, or dark corresponding to the visual response information. In addition, the experience information extraction system may extract the experience emotion information classified into any one emotion category of positive, negative, or neutral for product comparison reaction information or price reaction information.

After step S1012, the experience information extraction system calculates a sensitivity index for the experience target product by allocating an emotion score for each of the positive emotion category, the negative emotion category, and the neutral emotion category for the experience emotion information. Can (step S1014).

The weight applied to the emotion score may be set based on the frequency of occurrence of positive emotion words, negative emotion words, or neutral emotion words. The experience information extraction system assigns a unique integer index using the BoW algorithm for each text information corresponding to the experience emotion information classified by positive emotion category, negative emotion category, or neutral emotion category, and a word token at the position of each index. By recording the number of occurrences of the word, the frequency of occurrence of the word can be calculated. Thereafter, the experiential information extraction system reflects the weight set higher to the emotional score as the calculated frequency of occurrence of the word is higher, and reflects the weight set lower as the frequency of appearance to the corresponding emotional score. Accordingly, even with the same positive feeling, negative feeling, or neutral feeling, the intensity of positive feeling, negative feeling, or neutral feeling can be distinguished according to the frequency of appearance of the word.

After step S1014, the experience information extraction system calculates a potential purchase rate representing the potential purchase possibility of the experience target product by the experiencer using the calculated sensitivity index and the actual purchase index for the various experience target products of the experiencer. Can (S1016 step).

The experience information extraction system may calculate a potential purchase rate by applying a scaling value to at least one or more of the calculated sensitivity index and the skin suitability index for the beauty product when the product to be experienced corresponds to a beauty product. Here, the scaling value may be a value to which a sensitivity index or a skin fitness index is replaced with an reciprocal number, or a value to which a weight is assigned to a value substituted by an reciprocal number.

The experience information extraction system applies a scaling value to the actual purchase index, which is a value calculated through an arithmetic operation according to a combination of one or more of the number of visits of the experiencer, the number of purchases of the experiencer, and the purchase product of the experiencer, and the scaled actual purchase index The potential purchase rate can be calculated by arithmetically calculating and scaled sensitivity index or skin fitness index with each other.

In addition, when the experience target product corresponds to a beauty product, the experience information extraction system may extract a product recommended to the experiencer using the sensitivity index and skin characteristic information according to skin measurement for each body part of the experiencer. .

The experience information extraction system determines whether the calculated response index is above a certain threshold. The experience information extraction system may extract beauty product groups for each body part to which beauty products with a sensitivity index greater than or equal to a certain threshold from among various beauty product group information corresponding to skin characteristics for each body part, when the calculated sensitivity index is above a certain threshold. . Thereafter, the experience information extraction system may extract beauty products matching the previously extracted skin characteristic information from among the extracted beauty product groups for each body part.

After step S1016, the experience information extraction system may output one or more of the experience reaction information and the experience emotion information according to input of any one of the experience person identification information and the product identification information (step S1016).

The experience information extraction system may include an input module such as a keypad, a jog shuttle, a touch screen, and an input button in order to receive the user identification information or product identification information. In addition, the experience information extraction system may output experience response information, experience emotion information, sensitivity index, skin characteristic information, skin suitability information, or potential purchase rate information corresponding to the operation result of each component.

In addition, the experience information extraction system may include a module that displays image data captured by a camera module or a device management module related to image display to output skin characteristic information. For example, the experience information extraction system detects feature points corresponding to skin aging, including skin wrinkles and blemishes, based on analysis data including grayscale values, density, area, and linearity of pixels having a density higher than a reference value. I can. The experiential information extraction system can extract feature points from image data using feature point extraction algorithms such as the Speeded-Up Robust features (SURF) algorithm that accelerates the Scale Invariant Feature Transform (SIFT) algorithm.

In addition, the experience information extraction system converts the image data captured by the camera module into gray scale images or performs normalization processing according to a predetermined reference value, and performs image processing to remove various noises from the image data. Can be done. In addition, the experience information extraction system can remove noise in image data using an impulse noise removal filter, and a sharpening spatial filter is used to remove noise from image data from which noise is removed. It is possible to reinforce the resulting loss edge.

In addition, the experience information extraction system can manage and display skin images by dividing each part of the face (for example, forehead, neck, under the chin, glabellar, under the eyes, and around the nose and mouth). . In other words, the experience information extraction system uses skin aging information for each part of the face (e.g., skin age, aging graph) from feature points corresponding to skin aging positions for each part of the face as the skin image application is driven. ) Can be created and stored, and skin aging information for each part of the face can be displayed at the request of the user.

The above-described method may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

Software for extracting experience information may include a computer program, code, instruction, or a combination of one or more of them, and configure the processing device to operate as desired, or independently or combined It is possible to collectively command the processing unit. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

As described above, the embodiments disclosed in the specification of the present invention do not limit the present invention. The scope of the present invention should be interpreted by the following claims, and all technologies within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

10: Experience information extraction system
20: communication network
30: portable terminal
40: product tag
50: external terminal
100: storage
102: identification information recognition unit
104: text extraction unit
106: feature information extraction unit
108: reaction information extraction unit
110: emotional information extraction unit
112: control unit
114: response index calculation unit
116: skin characteristic detection unit
118: goodness-of-fit calculation unit
120: purchase rate calculation unit
122: recommended product extraction unit
124: user interface unit
126: Ministry of Communications
10: logistics service system
100: user terminal
110: terminal communication unit
120: terminal memory
130: terminal processor
200: logistics server
210: Ministry of Logistics and Communication

Claims (37)

A feature information extraction unit that performs pre-processing of text information generated according to the experience of the experience target product, and extracts feature information, which is an input value for machine running, from the pre-processed text information;
A reaction information extracting unit that applies the extracted feature information to a first model trained based on product group experience data according to a consumer experience, and extracts experience reaction information classified by category according to the experience of the experience target product;
A sentiment information extracting unit for extracting experience sentiment information by applying the feature information of each category of sentences of the experience response information extracted by the reaction information extracting unit to a second model; And
Matching the experience response information or the experience emotion information with the experience response information or the experience emotion information, and controlling to store the matched experience response information or the experience emotion information in a storage unit Including a control unit,
The feature information extraction unit,
Allocating sentence identification information for each of the divided sentences of the text information,
The reaction information extraction unit,
Using the first model, for each of the divided sentences corresponding to the sentence identification information, functional reaction information, tactile reaction information, visual reaction information, product comparison reaction information, and price reaction according to the experience of the experience target product Extracting the experience response information including at least one of the information,
The emotion information extraction unit,
Using the second model, experience information extraction system according to product experience, characterized in that the experience response information corresponding to the sentence identification information is classified into at least one of a positive emotion category, a negative emotion category, and a neutral emotion category. . The method according to claim 1,
Further comprising an identification information recognition unit for recognizing the user identification information or the product identification information,
The control unit controls the start of the operation of the reaction information extraction unit in response to recognition of the experience identification information or the product identification information of the identification information recognition unit,
Experience information extraction system according to product experience, characterized in that matching the experience response information extracted by the reaction information extraction unit with the experience person identification information or the product identification information. The method according to claim 1,
And a text extracting unit for extracting the text information of the experienced user from the voice of the experienced user generated according to the experience of the experience target product. delete The method according to claim 1,
The feature information extraction unit,
A pre-processing module for dividing the text information into sentences by using natural language program information for natural language processing, and performing text pre-processing for each of the divided sentences; And
And a feature vector calculation module that calculates any one feature vector of an integer encoding vector, a one-hot encoding vector, and an embedding vector from the text pre-processed split sentence. Experience information extraction system according to product experience. The method of claim 5,
The feature vector calculation module,
Experience information extraction system according to product experience, characterized in that the embedding vector is calculated using the product group corpus information processed by the corpus for natural language processing and experience information modeling of word information related to the experience product group. The method of claim 6,
The first model,
After the training data and test data included in the product group experience data are refined by the product group corpus information, the experience response information according to the experience of the experience target product is categorized through application of the refined training data and the test data. Experience information extraction system according to product experience, characterized in that modeled to classify by category. delete The method according to claim 1,
If the above experience target product is a beauty product,
The reaction information extraction unit,
Functional reaction information according to the experience of the beauty product, tactile reaction information including at least one of at least one of moisture, application, oil, stickiness, and residual sensation according to the experience of the beauty product, brightness and brightness according to the experience of the beauty product , At least one reaction of visual reaction information including at least one of heaviness and darkness, product comparison reaction information according to value comparison between the beauty product and other beauty products, and price reaction information on the price of the beauty product Experience information extraction system according to product experience, characterized in that classification by information category and extracting the experience response information. delete The method according to claim 1,
The first model or the second model,
Use any one of a recurrent neural network (RNN) model, a long short term memory networks (LSTM) model, a convolution neural network (CNN) model, a gated recurrent unit (GRU) model, and a transformer model,
To construct a deep neural network model, activate at least one of the Softmax function, Hyperbolic Tangent function, Sigmoid function, ReLU function, and Leaky ReLU function. Experience information extraction system according to product experience, characterized in that using a function. delete The method according to claim 1,
It characterized in that it further comprises a sensitivity index calculation unit for calculating a sensitivity index for the experience target product by allocating an emotion score for each of the positive emotion category, the negative emotion category and the neutral emotion category with respect to the experience emotion information. Experience information extraction system according to product experience. The method of claim 13,
It characterized in that it further comprises a purchase rate calculation unit for calculating a potential purchase rate indicating a potential purchase possibility for the product to be experienced by the experiencer by using the calculated sensitivity index and the actual purchase indices for various products subject to experience by the experiencer. Experience information extraction system according to product experience. The method of claim 14,
If the above experience target product is a beauty product,
The purchase rate calculation unit,
Applying a scaling value to at least one of the sensitivity index, the actual purchase index, and the skin suitability index of the beauty product according to the skin type and skin tone of the experiencer to calculate the potential purchase rate for the beauty product of the experiencer Experience information extraction system according to product experience, characterized in that. The method according to claim 1,
If the above experience target product is a beauty product,
Further comprising a skin characteristic detection unit for measuring the skin of each body part of the experiencer, and detecting skin characteristic information including at least one of skin type information and skin tone information for each body part of the experiencer,
The control unit,
According to a product experience, characterized in that the skin characteristic information is matched with the experience response information corresponding to the experience person identification information, and the skin characteristic information matched with the experience response information is stored in the storage unit. Experience information extraction system. The method of claim 16,
The storage unit stores beauty attribute information for beauty products among various experience target products,
The system for extracting experience information according to product experience, further comprising a fitness calculation unit for calculating a skin fitness index of the beauty product according to the skin type and skin tone of the experiencer, using the beauty attribute information and the skin characteristic information. The method of claim 17,
The suitability calculation unit,
The skin type attribute information of the experiencer's detected skin type information and the beauty attribute information of the beauty product are compared, or the skin tone attribute information of the experiencer's detected skin tone information and the beauty attribute information of the beauty product is compared Experience information extraction system according to product experience, characterized in that calculating a fitness index. The method of claim 13,
If the above experience target product is a beauty product,
And a recommended product extracting unit for extracting a product recommended to the experienced user by using the sensitivity index and skin characteristic information according to the skin measurement for each body part of the experiencer. In the experience information extraction method according to product experience performed by the experience information extraction system according to product experience,
Performing, by a feature information extracting unit, pre-processing of text information generated according to an experience of an experience target product, and extracting feature information, which is an input value for machine running, from the pre-processed text information;
Extracting, by a response information extracting unit, the extracted feature information to a first model trained based on product group experience data according to consumer experience, and extracting experience response information classified by category according to the experience of the experience target product;
Extracting experiential emotional information by applying the feature information for each category of the experiential response information extracted by the emotional information extracting unit to a second model; And
The control unit matches the experience response information or the experience emotion information with the experience response information or the experience emotion information, and stores the matched experience response information or the experience emotion information in a storage unit. Including steps,
Extracting the feature information,
Allocating sentence identification information for each of the divided sentences of the text information,
Extracting the experience response information,
Using the first model, for each of the divided sentences corresponding to the sentence identification information, functional reaction information, tactile reaction information, visual reaction information, product comparison reaction information, and price reaction according to the experience of the experience target product Extracting the experience response information including at least one of the information,
Extracting the experience emotional information,
Using the second model, the experience response information corresponding to the sentence identification information is classified into at least one of a positive emotion category, a negative emotion category, and a neutral emotion category. . The method of claim 20,
The identification information recognition unit further comprises the step of recognizing the user identification information or the product identification information,
After recognizing the user identification information and the product identification information, the characteristic information extraction unit extracts the characteristic information. The method of claim 20,
Extracting the feature information,
Dividing the text information into sentences by using natural language program information for natural language processing;
Performing text preprocessing for each of the divided sentences; And
And calculating a feature vector of any one of an integer encoding vector, a one-hot encoding vector, and an embedding vector from the text preprocessed split sentence. Method of extracting experience information according to. The method of claim 22,
The step of calculating the feature vector,
A method of extracting experience information according to product experience, characterized in that the embedding vector is calculated by using the product group corpus information processed by the corpus for natural language processing and experience information modeling of word information related to the experience product group. delete The method of claim 23,
The first model,
After training data and test data included in the product group experience data are refined by the product group corpus information, the experience response information according to the experience of the experience target product is categorized through application of the refined training data and the test data. A method of extracting experience information according to product experience, characterized in that modeled to classify each. The method of claim 20,
The text extracting unit further comprises the step of extracting the text information of the experienced person from the voice of the experienced person generated according to the experience of the experience target product,
After extracting the text information, the feature information extracting unit extracts the feature information. The method of claim 20,
Extracting the experience response information,
When the product to be experienced is a beauty product, functional reaction information according to the experience of the beauty product, a sense of moisture according to the experience of the beauty product, a tactile sense including at least one of a feeling of application, oil, stickiness, and residual feeling Response information, visual response information including at least one of brightness, brightness, heaviness, and darkness according to the experience of the beauty product, product comparison reaction information according to value comparison between the beauty product and other beauty products, and the beauty A method of extracting experience information according to product experience, characterized in that the experience response information is extracted by classifying by at least one or more reaction information categories among price response information on the price of a product. delete The method of claim 20,
The first model or the second model,
Use any one of a recurrent neural network (RNN) model, a long short term memory networks (LSTM) model, a convolution neural network (CNN) model, a gated recurrent unit (GRU) model, and a transformer model,
To construct a deep neural network model, activate at least one of the Softmax function, Hyperbolic Tangent function, Sigmoid function, ReLU function, and Leaky ReLU function. A method of extracting experience information according to product experience, characterized by using a function. delete The method of claim 20,
The step of calculating a sensitivity index for the experience target product of the experiencer by allocating an emotion score for each of the positive emotion category, the negative emotion category, and the neutral emotion category for the experience emotion information. Experience information extraction method according to product experience, characterized in that. The method of claim 31,
Further comprising the step of calculating, by a purchase rate calculation unit, a potential purchase rate representing a potential purchase possibility for the product to be experienced by the experiencer using the calculated response index and the actual purchase index for the various products to be experienced by the experiencer. A method of extracting experience information according to product experience as a feature. The method of claim 32,
If the above experience target product is a beauty product,
Applying a scaling value to at least one of the actual purchase index, the calculated sensitivity index, and the skin suitability index of the beauty product according to the skin type and skin tone of the experiencer to determine the potential purchase rate of the beauty product of the experiencer Experience information extraction method according to product experience, characterized in that to calculate. The method of claim 20,
If the above experience target product is a beauty product,
Detecting skin characteristic information including at least one of skin type information and skin tone information for each body part of the experiencer by measuring, by a skin characteristic detection unit, the skin for each body part of the experiencer; And
The control unit further comprises the step of matching and processing the skin characteristic information with the experience response information corresponding to the experienced person identification information, and storing the skin characteristic information matched with the experience response information in the storage unit. Method of extracting experience information according to product experience. The method of claim 34,
Further comprising the step of calculating, by the fitness calculation unit, the skin fitness index of the beauty product according to the skin type and skin tone of the experiencer, using the beauty attribute information and the skin characteristic information for each of the beauty products among various experience target products. Experience information extraction method according to product experience, characterized in that. The method of claim 35,
The step of calculating the skin fitness index,
The skin type attribute information of the experiencer's detected skin type information and the beauty attribute information of the beauty product are compared, or the skin tone attribute information of the experiencer's detected skin tone information and the beauty attribute information of the beauty product is compared Experience information extraction method according to product experience, characterized in that calculating a fitness index. The method of claim 31,
If the above experience target product is a beauty product,
And extracting, by the recommended product extraction unit, a product recommended to the experiencer using the sensitivity index and skin characteristic information according to skin measurement for each body part of the experiencer.

Images