KR20210017973A - Method and server for food ingredient pairing prediction using siamese neural network - Google Patents

Abstract

According to one aspect of the present invention, a food ingredient pairing prediction method using a Siamese neural network, which is conducted by a server, comprises: a step of collecting one or more pieces of cooking information through a network, extracting food ingredients for each piece of cooking information collected, and generating food ingredient data; a step of calculating a food ingredient pairing score in accordance with the simultaneous appearing number of times of food ingredients based on the food ingredient data and providing pairing data; and a step of providing the result data predicting the food ingredient pairing from a food ingredient pairing prediction model with the input of pairing data. The food ingredient pairing prediction model is trained by using the Siamese neural network made of a first subnetwork and a second subnetwork with the input data of the food ingredient data and the food ingredient pairing score data. The present invention is able to discover various food ingredient pairings.

Description

Food ingredient pairing prediction method and server using Siamese twins network {METHOD AND SERVER FOR FOOD INGREDIENT PAIRING PREDICTION USING SIAMESE NEURAL NETWORK}

The present invention relates to predicting food ingredient pairing, and more particularly, to a method and server for predicting food ingredient pairing using a Siamese twin network as input of various types of food ingredient data.

There are numerous ingredients in the cooking industry, and there are numerous ingredients, but chefs and culinary researchers use only a handful of ingredients.

In the food pairing field, combinations of various types of ingredients are recommended through food pairing guidebooks, etc., but since food pairing is based on the experience of experts, the subjective aspect is strong, making it difficult to quantify, and there are limitations in recommending new recipes .

One of the food pairing studies introduces a flavor network in which the edge of the network is built based on the number of flavor compounds shared by food ingredients. The flavor network consists of 381 components and 1,021 flavor compounds. FlavorDB combines existing food repositories to provide a larger database with user interactive pages. Food bridging improves the flavor network by adding a bridge between the two ingredients through a pair of affinity, despite the poor chemical similarity of the two ingredients. However, they contain only a limited number of flavor compounds and natural ingredients, but some well-known food pairs (eg, red wine and beef) have few flavor compounds in common.

Although artificial intelligence is recently applied to many domains, the application of artificial intelligence to data dealing with food information is still insufficient.

Republic of Korea Patent Publication No. 10-2018-0086712 (Name of invention: IoT-based food ingredient inquiry and recipe content provision platform)

The present invention provides a method of predicting food ingredient pairing using an artificial neural network-based model as to solve the above-described problems of the prior art.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for solving the above-described technical problem, the method for predicting food ingredient pairing using the Siamese twin network performed by the server according to the first aspect of the present disclosure is to collect one or more cooking information through the network, and Generating food material data by extracting food ingredients for each information; Providing pairing data by calculating a food ingredient pairing score according to a frequency of simultaneous appearance of ingredients based on the food ingredient data; And providing data as a result of predicting food ingredient pairing in the food ingredient pairing prediction model by inputting pairing data. The food ingredient pairing prediction model is trained by using food ingredient data and ingredient pairing score data as input data and using a Siamese twin network composed of a first sub-network and a second sub-network.

As a technical means for solving the above technical problem, a food ingredient pairing prediction server using a Siamese twin network according to a second aspect of the present disclosure includes: a memory storing a food ingredient pairing prediction program using a Siamese twin network; It includes a processor that executes a program stored in the memory, and the processor collects one or more cooking information through a network by execution of the program, extracts food ingredients for each collected cooking information, and generates food material data, based on the food material data. Thus, the pairing data is provided by calculating the food ingredient pairing score according to the frequency of simultaneous appearance between ingredients, and the pairing data is input to provide the result data of predicting food ingredient pairing in the food ingredient pairing prediction model. The food ingredient pairing prediction model is trained by using food ingredient data and ingredient pairing score data as input data and using a Siamese twin network composed of a first sub-network and a second sub-network.

According to any one of the above-described problem solving means of the present application, a new and diverse food ingredient pairing can be discovered by utilizing an artificial neural network-based Siamese twin network for food ingredient pairing prediction.

By constructing a large-scale data set including various food ingredient pairings, more data can be analyzed than food ingredient pairing recommendations by experts, and complementary ingredient pairings can be recommended through qualitative analysis, and new ingredient pairings can be discovered.

1 is a block diagram showing the configuration of a food ingredient pairing prediction server using a Siamese twin network according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a food ingredient pairing prediction system using a Siamese twin network according to an embodiment of the present invention.
Figure 3 is a flow chart showing the progress of the food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.
4 is a diagram illustrating a food ingredient pairing score in a method of predicting food ingredient pairing using a Siamese twin network according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating the architecture of a food ingredient pairing prediction model in a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.
6 is a diagram illustrating a prediction result of a model in a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.
7 is a diagram illustrating a result of an ablation test for a method of predicting food ingredient pairing using a Siamese twin network according to an embodiment of the present invention.
8 is a diagram illustrating a known pairing and a new pairing result in a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.
9 is a diagram illustrating a pairing ranking in a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.
10 is a diagram illustrating a pairing ranking of a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present application. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts not related to the description are omitted in order to clearly describe the present application, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" with another part, this includes not only the case that it is "directly connected", but also the case that it is "electrically connected" with another element interposed therebetween. do.

Throughout this specification, when a member is positioned “on” another member, this includes not only the case where a member is in contact with another member, but also the case where another member exists between the two members.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the configuration of a food ingredient pairing prediction server using a Siamese twin network according to an embodiment of the present invention.

As shown, the food ingredient pairing prediction server 100 using a Siamese twin network may include a communication module 110, a memory 120, a processor 130, a database 140, and an input module 150.

The communication module 110 communicates data with the connected user terminal 100 and the linked server 200, respectively. The communication module 110 may be a device including hardware and software necessary to transmit and receive signals such as control signals or data signals through wired/wireless connection with other network devices.

The memory 120 stores a food ingredient pairing prediction program using the Siamese twin network. The food ingredient pairing prediction program collects one or more cooking information through a network, extracts food ingredients by the collected cooking information to generate food material data, and calculates a food ingredient pairing score according to the frequency of simultaneous appearances between ingredients based on the food material data. Pairing data is provided, and the result data of predicting food ingredient pairing in a food ingredient pairing prediction model is provided by inputting pairing data. In this case, the food ingredient pairing prediction model is trained using the food ingredient data and the ingredient pairing score data as input data, and using a Siamese twin network composed of a first sub-network and a second sub-network.

The memory 120 stores various types of data generated during execution of an operating system for driving the food ingredient pairing prediction server 100 using the Siamese twins network or the food ingredient pairing prediction program using the Siamese twins network.

In this case, the memory 120 collectively refers to a nonvolatile storage device that continuously maintains stored information even when power is not supplied and a volatile storage device that requires power to maintain the stored information.

In addition, the memory 120 may perform a function of temporarily or permanently storing data processed by the processor 130. Here, the memory 120 may include a magnetic storage media or a flash storage media in addition to a volatile storage device requiring power to maintain stored information, but the scope of the present invention is limited thereto. It does not become.

The processor 130 executes the program stored in the memory 140, but controls the entire process according to the execution of the food ingredient pairing prediction program using the Siamese twin network. Each operation performed by the processor 130 will be described in more detail later.

The processor 130 may include all types of devices capable of processing data. For example, it may refer to a data processing device embedded in hardware having a circuit physically structured to perform a function represented by a code or command included in a program. As an example of a data processing device built into the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, and an application-specific ASIC ( Integrated circuit), a field programmable gate array (FPGA), and the like may be covered, but the scope of the present invention is not limited thereto.

The database 140 stores or provides data necessary for a food ingredient pairing prediction system using a Siamese twin network under the control of the processor 130. The database 140 may be included as a separate component from the memory 140 or may be built in a partial area of the memory 140.

2 is a block diagram showing the configuration of a food ingredient pairing prediction system using a Siamese twin network according to an embodiment of the present invention.

The used food ingredient pairing prediction system includes a server 100 and a plurality of user terminals 200. They can be connected through a network.

A method for predicting food ingredient pairing using a Siamese twin network may be implemented in the server 100. In the food ingredient pairing prediction method using the Siamese twins network, data analysis may be performed in the server 100 and the analysis data may be displayed in the user terminal 200.

Figure 3 is a flow chart showing the progress of the food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.

In the method of predicting food ingredient pairing using the Siamese twins network, one or more cooking information is collected through the network (S110), and food ingredients are extracted by extracting food ingredients for each of the collected cooking information (S120).

When extracting food ingredients, the processor 130 may extract the food ingredient name by performing logistic regression on each word from the collected cooking information using a food ingredient name extraction module based on a recurrent neural network. In this case, the recurrent neural network can use bidirectional Long Short Term Memory models (LSTM).

For example, you can take advantage of Recipe1M, a data set containing about a million recipes and their images, collected from several popular websites related to cooking. All contents of Recipe1M can be divided into two categories: text and images. Recipe1M's recipe text consists of two parts: an ingredient list and recipe instructions. Im2Recipe uses a bidirectional LSTM-based ingredient name extraction module to perform logistic regression on each word from Recipe1M's list of all ingredients to extract only ingredient names. For example, "2 tbsp of olive oil" is extracted with olive oil. In the instructions for the recipe, all vocabulary, including the names of ingredients, can be learned in a word-to-vector (word2vec) fashion.

For example, as shown in Table 1 below, 3,567 unique material names can be obtained out of a total of 30,167 learned vocabularies. Known pairs consist of ingredients with an occurrence number of greater than 20. Known pairs have at least five or more simultaneous occurrences.

[Table 1]

After the food material data are generated, the food material pairing score is calculated based on the food material data according to the frequency of simultaneous appearances between the food ingredients, and the pairing data is provided (S130).

Traditional ingredient pairing methods rely on extensive experience and knowledge of human experts in the culinary industry. As a result, there is not enough data available to train a deep learning model. To solve this problem, we construct a new large-scale food ingredient pairing score data set that can train deep learning models. Food pairings are scored from -1 to 1 depending on how well the ingredients complement each other. Co-occurrence information of ingredients in a large number of recipe corpus can provide insight into how ingredients are combined in each recipe. In this study, the number of ingredients or the number of cooking procedures was not considered because the data set contained statistical co-occurrence information.

The pairing data is obtained by calculating and normalizing the food ingredient pairing score based on the NPMI (normalized point-wise mutual information), which calculates the probability that two ingredients occur individually and the probability that they occur simultaneously in the cooking information. I can.

[Equation 1]

p(x,y)=

p(x)=

p(y)=

In the present invention, a food ingredient pairing score is calculated based on point-wise mutual information (PMI). The PMI score indicated in Equation 1 is the probability of two elements in which p (x, y) occurs simultaneously, which is compared with the probability of each element in which p (x) and p (y) occurs individually. Custom scores are designed to accurately represent good/bad pairs by penalizing very popular ingredients such as salt or butter with low co-occurrence pairs. On the other hand, pairs made of less popular ingredients can be good complementary pairs (eg Wasabi & Nori). In order to better train and fit the regression model, we can use the normalized version of the PMI (NPMI) shown in Equation 1 above. The point-unit mutual information can be normalized between -1 and +1. Here, -1 (limit) never occurs together, 0 represents independent, +1 represents complete simultaneous occurrence. So, with a score between -1 and 1, you can intuitively determine if a pair is suitable.

When creating a food pairing data set based on the NPMI score, ideally, you can calculate all 6,359,961 possible food pairing scores (3,567 2) generated from 3,567 unique ingredients. However, foods that rarely occur in one million recipe texts can serve as noise samples, and foods that rarely occur can degrade the performance of the model. Therefore, pairings were established by removing food ingredients whose occurrence frequency does not exceed 20 and food ingredients whose number of occurrences does not exceed five. The results can be referred to Table 1 above. A total of 356,451 valid known pairings were obtained, and all other pairings were considered unknown pairings. The final distribution of food pairing scores follows a roughly normal distribution. In the top 5% of the distribution (μ + 2σ), the material pairing score (0.274 or higher) is the complement pair, and the score with a negative number can be assumed to be an example of poor pairing, as shown in Table 2.

[Table 2]

Among the material pairing data sets, the five best ingredients in the data set for pairing with vanilla are shown in Table 2. In accordance with expert judgment, The Flavor Bible, it is also recommended to use chocolate, coffee, cream, ice cream or sugar with vanilla.

After calculating the food ingredient pairing score, the processor 130 may build a food ingredient pairing score database using one or more normalized ingredient pairing scores.

The pairing data is input and the result data of predicting food ingredient pairing in the food ingredient pairing prediction model is provided (S140). In this case, the food ingredient pairing prediction model may be learned by using the food ingredient data and the food ingredient pairing score data as input data, and using a Siamese twin network composed of a first sub-network and a second sub-network.

The food ingredient pairing prediction model updates weights to calculate food ingredient pairing scores for the first ingredient and the second ingredient, and the ingredient pairing score may be calculated through a deep layer for semantic features and a wide layer for sparsity features.

The food ingredient pairing prediction model calculates the loss function (J(θ)) using the actual food ingredient pairing score (y _ab ) and the predicted food ingredient pairing score (Y _ab ) using the cosine similarity by Equation 2 below, and the result of the loss function It may be to determine the optimal weight through learning so that the value is minimized.

[Equation 2]

N: Total number of inputs used for training

Equation 2 is for minimizing the loss function (mean squared error), L is the computational loss function to be minimized during training, Θ is the model parameter to be trained, and N is the total number of training data for the input pair, respectively. yab is the actual score value, and Yab is the predicted score value. The proposed food ingredient pairing prediction model can use the Adam optimizer. Adam optimizer uses momentum to improve accuracy, and RMSprop to compensate for stride sensitivity, and can speed up learning quickly and stably.

The processor 130 may generate graph information consisting of a node representing food ingredients and an edge representing the degree of relationship between food ingredients and food ingredients, and the food ingredient pairing prediction model is a graph-based neural network architecture to learn one-to-many food ingredient pairing. You can use

The result data of predicting food ingredient pairing may include an ingredient pairing ranking list based on the ingredient pairing score, and may include an ingredient pairing score included in the cooking information, a new ingredient pairing score, and recommended ingredient pairing data.

4 is a diagram illustrating a food ingredient pairing score in a method of predicting food ingredient pairing using a Siamese twin network according to an embodiment of the present invention.

By training on datasets that include well-known ingredient pairings (e.g., Gin & Tonic Water, Salt & Pepper, Vanilla & Onion), the ingredient pairing predictive model is less popular or less frequently used and unknown ingredient pairing scores (e.g.: Gin & Aqua Beet, Wasabi & Nori, Lime & Nopal) can be predicted.

FIG. 5 is a diagram illustrating the architecture of a food ingredient pairing prediction model in a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.

Siamese twins networks can be used for a variety of tasks to learn the similarity between two different inputs, and some variations of the Siamese twins network can be used. In the present invention, a Manhattan LSTM model that receives two sentences as input may be used. The Manhattan LSTM model generates vector representations for each sentence and computes the similarity between vector representations using a simple function exp(h1 h2). Where h1 and h2 are embedding vectors. In addition, a custom contrast loss function, which can be divided into a partial loss function for a positive pair and a partial loss function for a negative pair, is available. The loss function widens the distance between two vector representations of a negative pair and narrows the distance between two vector representations of a positive pair. Unlike these two functions using a Siamese twin network and accepting two different inputs of the same type, it can also take one input of a Siamese twin network model and use it as a standard to evaluate the other. The input is evaluated based on its similarity to the input used as the standard. The present invention is designed to train the semantic relationship of food ingredient pairing beyond the simple distance-based similarity mentioned above.

The architecture of the food pairing prediction model includes two main components. The first is a food ingredient expression component (module) using the Siamese twins network. The same two identical Multi-Layer Perceptron (MLP) weights each receive a different 300-dimensional word vector representation. Each MLP has two fully connected layers that process the input material vector. Considering (X _a , X _b ) as one food ingredient pairing expressed as a 300-dimensional word vector, W _* and b _* are the shared weights and biases of MLP, respectively, and f (·) represents the activation function for nonlinearity. We use f (·) (max (x, 0)) as the rectified linear unit (ReLU). The learned expression (h _a , h _b ) of this pair is mathematically expressed as Equation 3 below.

[Equation 3]

이고 i 및 j는 숨겨진 유닛의 수이다.

And i and j are the number of hidden units.

The'pairing score prediction component (module)' uses wide&deep learning. Layers are divided into wide layers and deep layers. In the deep layer, two j-dimensional learning representation vectors are concatenated and passed to another MLP that computes the joint representation of the two components. This joint expression is expressed as a dip vector d and is mathematically expressed as in Equation 4 below.

[Equation 4]

이고 j는 각 레이어의 숨겨진 유닛의 수. 와이드 레이어에서, 2 개의 j 차원 학습 표현 벡터의 외부 곱이 계산되고 j × j 행렬이 생성된다. 행렬은 하기 수학식5와 같이 수학적으로 표현되는 j 2 차원의 와이드 벡터 w로 평탄화된다.

And j is the number of hidden units in each layer. In the wide layer, the outer product of two j-dimensional learning expression vectors is calculated and a j × j matrix is generated. The matrix is flattened into a j2-dimensional wide vector w expressed mathematically as in Equation 5 below.

[Equation 5]

Where g denotes a flattening function that transforms an n × n matrix into a single vector of n ² dimensions.

Then the wide vector w is directly connected to the deep vector d. The concatenation of the wide vector and the deep vector is finally passed to the fully connected layer to calculate the pairing score for the final output. Overall, since (w, d) is a connection between the wide vector and the deep vector, the final output score Y for the food ingredient pair (Xa, Xb) is mathematically calculated as in Equation 6 below.

[Equation 6]

및

이고, j는 각 레이어에서 숨겨진 유닛의 수이다.here

And

And j is the number of hidden units in each layer.

In the food ingredient pairing prediction model, the loss function (J(θ)) using the predicted food ingredient pairing score (Y _ab ) using the actual food ingredient pairing score (y _ab ) and the cosine similarity was calculated by Equation 7 below, and the result of the loss function It may be to determine the optimal weight through learning so that the value is minimized.

[Equation 7]

Equation 7 is for minimizing the loss function (mean squared error), L is the calculation loss function to be minimized during training, Θ is the model parameter to be trained, and N is the total number of training data for the input pair, respectively. yab is the actual score value, and Yab is the predicted score value. The proposed food ingredient pairing prediction model can use the Adam optimizer. Adam optimizer uses momentum to improve accuracy, and RMSprop to compensate for stride sensitivity, and can speed up learning quickly and stably.

6 is a diagram illustrating a prediction result of a model in a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.

Before evaluating the food pairing prediction model, the base model can be evaluated first. First, the pairing score is predicted by calculating the cosine similarity between the two input component vectors. Then, we use the machine learning model equivalent of the Python Scikit-learn package, a linear support vector regression analyzer, an arbitrary forest regression analyzer, an additional tree regression analyzer, an SGD regression analyzer, and gradient boosting as the base model. It also uses a simple version of the Siamese twins network in one of the base models. All of these models are equipped with hyperparameters estimated by built-in grid search.

The following five metrics are used to evaluate model performance as described in Table 3 below. Compared with RMSE (Root Mean Square Error), MSE (Mean Square Error), MAE (Mean Absolute Error), CORR (correlation) and R2, the food ingredient pairing prediction model (KitcheNette) of the present invention performs better than the basic model in all metrics. This is excellent.

[Table 3]

The ranking performance of the model was evaluated using NDCG @ K (Normalized Discounted Cumulative Gain) (Fig. 6a) and the sensitivity of the model was evaluated using the ROC curve (Fig. 6b). From NDCG @ K's point of view, our model outperforms all basic models in making accurate predictions. The ROC curve is also used to measure the classification performance of the model. All pairings with a predicted score of 0.274 or higher are considered as complementary pairings. Pairing with a low score is considered non-complementary. The ROC curve results show that the food ingredient pairing prediction model of the present invention achieves higher performance than all other models in the complementary pairing classification.

7 is a diagram illustrating a result of an ablation test for a method for predicting food ingredient pairing using a Siamese twin network according to an embodiment of the present invention.

Ablation tests are performed to evaluate each function of the present invention. As can be seen from FIG. 7, the inclusion of extensive and in-depth architectures and components of the food ingredient pairing prediction model of the present invention helps to improve performance. When the food ingredient pairing prediction model of the present invention uses the cosine similarity of the expression learned in the Siamese twin network, the lowest performance is obtained in the food ingredient pairing score prediction. The connection of the two expressions (deep layer) greatly improves the performance of the model. This indicates that semantic relationships must be learned to predict food ingredient pairing scores. In addition, the wide deep architecture that learns the relationship between the two components further improves the model's performance. Also, using component embeddings in the input vectors instead of randomly initialized vectors improves the model's performance.

8 is a diagram illustrating a known pairing and a new pairing result in a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.

In order to verify the accuracy of the prediction of the food ingredient pairing prediction method using the Siamese twin network for the food ingredient pairing that is not frequently used, the prediction results of the known ingredient pairing and the unknown ingredient pairing were compared and analyzed. As shown in FIG. 8, after selecting three similar carbonated white wines (champagne, sparkling wine and prosecco), the score of each wine paired with other ingredients was calculated, and all possible pairings were made for the following three cases. Analyzed.

Case 1. We used champagne and orange twist, which are well-known ingredients that are already highly regarded, as a pairing for comparison. Orange Twist & Sparkling Wine and Orange Twist & Pro Seco pairings are rare, so there is no pairing score. We also chose two other ingredients (orange wedges and lime twist) that are similar to the orange twist, but aren't used often among the three wines. As a result, there is a pair of known pairings and eight unknown pairings. The predicted results of all 9 pairings were consistently high (0.33-0.45).

Case 2. Based on the calculated pairing scores, we simultaneously supplemented the wine with different ingredients to create three unique pairings: Champagne and Elderflower Liqueur, Sparkling Wine & Creme de Cassis, and Prosecco & Lemon Sorbet. The predicted outcome of the remaining 6 unknown pairings was consistently high compared to the 3 known known pairings (0.29-0.42).

Case 3. Finally, we made the worst pairing with champagne and paired with the other two wines to select onions. The predicted results of the two unknown pairings (sparkling wine & onion and prosecco & onion) were consistently as low as the score of the known pairing (champagne & onion).

In conclusion, it is shown that the food ingredient pairing prediction method using the Siamese twin network can predict a specific pairing based on analogical inference. This indicates that if A is similar to B and A forms a good pairing with C, then B is likely to form a good pairing. Using this inference, it can be considered that the performance of the method of predicting food pairing using the Siamese twin network in food pairing for which scores are not calculated.

9 is a diagram illustrating a pairing ranking in a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.

A comparative analysis is performed between ingredients graded by the food pairing prediction method (KitcheNette) using the Siamese twins network and ingredients graded in FlavorDB 4. Four popular food ingredients (tomato, onion, pepper and cinnamon) are selected and the top 10 ingredient pairs are searched for each selected ingredient as shown in Fig. 12. According to the present invention, food ingredients commonly used in daily cooking and meals (eg, tomatoes and lettuce, onions and ground beef, pepper and oregano, cinnamon and cloves, apples) are recommended. On the other hand, FlavorDB recommended selected ingredients and food ingredients that share many compounds, but some recommendations did not fit well with ingredients selected for cooking and eating (eg, tomatoes and tea, onions and cocoa, peppers and oranges).

10 is a diagram illustrating a pairing ranking of a food ingredient pairing prediction method using a Siamese twin network according to an embodiment of the present invention.

A new food-beverage pairing can be found according to the method of predicting food ingredient pairing using the Siamese twin network. Recommendations of “The Flavor Bible” and “WHAT to DRINK with WHAT you EAT”, which are specialized content for food and beverage recommendations, were displayed in the recommendations of FIG. 13 and compared with the present invention (KitcheNette). According to the method of predicting food ingredient pairing using the Siamese twins network, it not only provides recommendations that match the recommendations of culinary experts in this book, but also recommends far more pairings than two books.

For red and white wines, the present invention recommended a variety of meats (eg beef, lamb) and specific seafood ingredients (eg mussels, lobster, shrimp), respectively. The present invention recommended authentic Japanese food ingredients paired with sake, which shows that data-driven models can recommend less common food ingredients in Asian cuisine.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described in connection with specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The foregoing description of the present application is for illustrative purposes only, and those of ordinary skill in the art to which the present application pertains will be able to understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present application.

100: server
110: communication module
120: memory
130: processor
140: database
150: input module
200: user terminal

Claims (12)

In the food ingredient pairing prediction method using a Siamese twin network performed by a server,
(a) collecting one or more cooking information through a network, extracting food ingredients according to the collected cooking information, and generating food material data;
(b) providing pairing data by calculating a food ingredient pairing score according to the frequency of simultaneous appearance of ingredients based on the food ingredient data; And
(c) providing data as a result of predicting food ingredient pairing in a food ingredient pairing prediction model by inputting the pairing data; including,
The food ingredient pairing prediction model is learned by using the food ingredient data and the ingredient pairing score data as input data, and using a Siamese twin network consisting of a first sub-network and a second sub-network,
Food ingredient pairing prediction method using Siamese twins network. The method of claim 1,
When extracting food ingredients in step (a), using a food ingredient name extraction module based on a recurrent neural network, performing logistic regression on each word from the collected cooking information to extract food ingredient names,
Food ingredient pairing prediction method using Siamese twins network. The method of claim 2,
The recurrent neural network is based on bidirectional Long Short Term Memory models (LSTM),
Food ingredient pairing prediction method using Siamese twins network. The method of claim 1,
The pairing data is normalized by calculating a food ingredient pairing score based on the NPMI (normalized point-wise mutual information), which calculates the probability that two ingredients occur individually and the probability that they occur simultaneously in the cooking information. One,
Food ingredient pairing prediction method using Siamese twins network.
[Equation 1]

p(x,y)=

p(x)=

p(y)=

The method of claim 4,
In the step (b), comprising the step of building a food ingredient pairing score database using the one or more normalized food ingredient pairing scores,
Food ingredient pairing prediction method using Siamese twins network. The method of claim 1,
The food ingredient pairing prediction model updates weights to calculate the food ingredient pairing score for the first ingredient and the second ingredient, and calculates the ingredient pairing score through a deep layer for semantic features and a wide layer for sparsity features. sign,
Food ingredient pairing prediction method using Siamese twins network. The method of claim 6,
The food ingredient pairing prediction model calculates a loss function (J(θ)) using an actual food ingredient pairing score (y _ab ) and a predicted food ingredient pairing score (Y _ab ) using the cosine similarity by Equation 2 below, and the loss function The optimal weight is determined through learning so that the result value of is minimized.
Food ingredient pairing prediction method using Siamese twins network.
[Equation 2]

N: Total number of inputs used for training The method of claim 1,
After the step (a), the step of generating graph information consisting of a node representing a food ingredient and an edge representing a degree of relation between the food ingredient and the food ingredient,
The food ingredient pairing prediction model uses a graph-based neural network architecture to learn one-to-many food ingredient pairing,
Food ingredient pairing prediction method using Siamese twins network. The method of claim 1,
The result data of predicting the food ingredient pairing includes a food ingredient pairing ranking list based on the food ingredient pairing score,
Food ingredient pairing prediction method using Siamese twins network. The method of claim 1,
The result data of predicting the food ingredient pairing includes food ingredient pairing score, new ingredient pairing score, and recommended ingredient pairing data included in the cooking information,
Food ingredient pairing prediction method using Siamese twins network. In a food ingredient pairing prediction server using a Siamese twin network,
A memory storing a food ingredient pairing prediction program using a Siamese twin network;
And a processor executing a program stored in the memory, wherein the processor executes the program,
Collecting one or more cooking information through a network, extracting food ingredients for each of the collected cooking information to generate food material data,
Provide pairing data by calculating a food ingredient pairing score according to the frequency of simultaneous appearance between ingredients based on the food ingredient data,
Provides data as a result of predicting food ingredient pairing in the food ingredient pairing prediction model by inputting the pairing data,
The food ingredient pairing prediction model is learned by using the food ingredient data and the ingredient pairing score data as input data, and using a Siamese twin network consisting of a first sub-network and a second sub-network,
Food ingredient pairing prediction server using Siamese twins network. A non-transitory computer-readable recording medium on which a computer program for performing the method according to claim 1 is recorded.

Images