KR20240087937A - Method and system for ensemble of recurrent neural network model - Google Patents

Abstract

The present disclosure provides a method for ensemble of recursive neural network models, performed by at least one processor. This method includes the steps of receiving an input sequence, receiving a plurality of recursive neural network models to process the input sequence, and ensembleing the outputs of the plurality of recursive neural network models at each time step to produce an output corresponding to the input sequence. It includes steps of generating a sequence and outputting an output sequence.

Description

Ensemble method and system for recursive neural network model {METHOD AND SYSTEM FOR ENSEMBLE OF RECURRENT NEURAL NETWORK MODEL}

The present disclosure relates to a method and system for ensemble of a recursive neural network model, and specifically, to a recursive neural network model that ensembles the outputs of a plurality of recursive neural network models at every time step to generate an output sequence corresponding to the input sequence. It relates to ensemble methods and systems.

Ensemble is a type of machine learning algorithm that creates and combines multiple classifiers to perform better than a single classifier. Ensemble learning, which trains multiple weak classifiers in harmony to obtain more accurate predictions than a single strong classifier, can be an effective way to improve model performance. A variety of ensemble learning methods are being developed using traditional machine learning algorithms. For example, bagging (i.e. bootstrap aggregation) is an ensemble learning technique that helps improve the performance and accuracy of machine learning algorithms. The way bagging aggregates the results of multiple models can vary depending on the problem being solved. By adopting a bagging ensemble strategy, the classification model can vote on individual predictions, and the regression model can average the output. Afterwards, the model can select the final result.

Boosting, one of the machine learning algorithms widely used in ensemble learning, unlike bagging, which trains in parallel, trains multiple models sequentially and updates the final classifier sequentially. When the starting model makes a prediction, the starting model can repeat the process of giving weight to incorrect data and focusing on incorrectly predicted data so that the next learning model can learn well. Additionally, boosting can use voting to select the final result.

Afterwards, various ensemble methods of bagging and boosting emerged and were applied to various tasks. Typically, ensemble methods such as RandomForest, Adaboost, XGBoost, and LightGBM showed excellent performance in classification and regression even in deep learning models. These ensemble methods can achieve generalization performance better than that of a single model. Additionally, heterogeneous ensembles can combine models of different structures across homogeneous environments. In particular, the structure of CNN (Convolution Neural Network) is suitable for traditional ensemble methods, and research is being actively conducted on various ensemble methods applied to computer vision fields and later fields such as speech recognition and text classification.

An ensemble can be used not only in CNN but also in RNN (Recurrent Neural Network) with a recursive structure to increase the accuracy of the final result by combining N prediction values. Ensemble methods using RNN-based models have been continuously studied until recently. Additionally, various ensemble methods have been proposed, including sequence-to-sequence, which consists of an RNN-based encoder-decoder. Ensemble methods for predicting time series data, such as weather and stock price data, may use a combination of RNN variants rather than a single model to determine the output. Other methods may use a combination of boosting algorithms such as AdaBoost and RNN variants to determine the final model. In this way, ensemble research on sequence-to-sequence, which determines the output by combining multiple models, is in progress, such as the ensemble method for predicting time series data.

The performance of the existing RNN ensemble method has been greatly improved, but it has the following limitations. First, in the final ensemble stage, only the final results of the model can be aggregated. Second, the recursive structure of the RNN may not be reflected. RNN performed well in the sequential data domain, but the structural advantages of RNN may not be reflected if only the final result is considered.

Specifically, RNNs may be suitable for processing sequential data (e.g., text, voice, video, etc.) and time-series data measured in time (e.g., weather, stock prices, etc.). However, although RNNs have a recursive structure in which previous predictions can affect the next predictions, traditional ensemble methods have a problem in that they do not reflect the structural properties of RNNs.

The present disclosure provides a method and system (device) for ensemble of recursive neural network models to solve the above problems.

The present disclosure may be implemented in various ways, including as a method, device (system), or computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, the ensemble method of a recursive neural network model includes the steps of receiving an input sequence, receiving a plurality of recursive neural network models to process the input sequence, and generating a plurality of recursive neural network models at each time step. It may include generating an output sequence corresponding to the input sequence by ensembleing the output of the neural network model and outputting the output sequence.

According to an embodiment of the present disclosure, a plurality of recursive neural network models may each be trained with different learning data.

According to one embodiment of the present disclosure, the plurality of recursive neural network models may be generative models.

According to an embodiment of the present disclosure, the plurality of recursive neural network models may be a sequence-to-sequence model.

According to an embodiment of the present disclosure, generating the output sequence includes generating outputs of a plurality of recursive neural network models in a first time step, and forming an agreement based on the outputs of the plurality of recursive neural network models in the first time step. It may include determining a settled output and removing a recursive neural network model with an output different from the agreed upon output.

According to an embodiment of the present disclosure, the agreed output may be determined based on the number of models having the same output among a plurality of recursive neural network models in the first time step.

According to an embodiment of the present disclosure, a recursive neural network model that is not removed at every time step may use the output it generated in the previous time step as the input of the current time step.

According to an embodiment of the present disclosure, the step of generating an output sequence includes generating an output sequence corresponding to the input sequence by connecting the output at each time step of the at least one recursive neural network model remaining in the last time step. More may be included.

According to an embodiment of the present disclosure, generating the output sequence includes generating outputs of a plurality of recursive neural network models in a first time step, and forming an agreement based on the outputs of the plurality of recursive neural network models in the first time step. It may include determining an output and generating outputs of a plurality of recursive neural network models by using the outputs agreed upon in the second time step as inputs to all of the plurality of recursive neural network models.

According to one embodiment of the present disclosure, the agreed output may be determined based on reliability values of the outputs of the plurality of recursive neural network models at the first time step.

According to an embodiment of the present disclosure, generating the output sequence may further include generating an output sequence corresponding to the input sequence by concatenating the agreed outputs at all time steps.

In order to execute the method according to an embodiment of the present disclosure on a computer, a computer program stored in a computer-readable recording medium may be provided.

An information processing system according to an embodiment of the present disclosure, comprising a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and at least one program Receives an input sequence, receives a plurality of recursive neural network models to process the input sequence, ensembles the outputs of the plurality of recursive neural network models at each time step, generates an output sequence corresponding to the input sequence, and outputs the output sequence. It may include commands for:

According to various embodiments of the present disclosure, a recursive network architecture provides an output at each time step, and that output can recursively affect the next output at the next time step. Accordingly, the ensemble method of the recursive neural network model can recursively consider the output of the entire model for each time step. Additionally, because it is relatively independent of the network model, it can be universally applied to various types of recursive network models.

According to various embodiments of the present disclosure, through the survival ensemble model, incorrect answer models that affect the final result may be filtered out in advance. Accordingly, the prediction performance of the model can be improved by reflecting the structural properties of the RNN.

According to various embodiments of the present disclosure, the consensus output can be fed back to the loser model without eliminating the loser model through the consensus ensemble model. Accordingly, the performance of the ensemble model can be improved because all models participate at every time step.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as “a person skilled in the art”) in the technical field to which this disclosure pertains from the description of the claims. It will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.
1 shows an example of an ensemble method of a recursive neural network model according to an embodiment of the present disclosure.
Figure 2 is a diagram showing the structure of a traditional majority voting ensemble model.
Figure 3 is a diagram showing an example process of a traditional majority vote ensemble model.
Figure 4 is a diagram showing the structure of a survival ensemble model according to an embodiment of the present disclosure.
Figure 5 is a diagram showing an example process of a survival ensemble model according to an embodiment of the present disclosure.
Figure 6 is a diagram illustrating an example process of a consensus ensemble model according to an embodiment of the present disclosure.
Figure 7 is a diagram illustrating an example process of a consensus ensemble model according to an embodiment of the present disclosure.
Figure 8 is a diagram showing a graph comparing the performance of the ensemble model of the present disclosure, a traditional majority voting ensemble model, and a single model.
9 is a flowchart illustrating an example of a method according to an embodiment of the present disclosure.
Figure 10 is a schematic diagram showing a configuration in which an information processing system is connected to communicate with a plurality of user terminals in order to generate an output sequence according to an embodiment of the present disclosure.
Figure 11 is a block diagram showing the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In the present disclosure, 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated with a server device and a cloud device configured together.

In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A, or may refer to each of some components included in a plurality of A. .

Ensemble methods utilize multiple machine learning models to achieve better prediction performance than single composition models. In most ensemble methods, each component model can communicate training data and output results. Accordingly, the ensemble method can be applied to various types of machine learning models. However, when applying ensemble methods to a recursive network model, the output of the recursive network model may typically be a set of outputs rather than one of several labels. The degree of diversity in these outputs is so high that consensus achieved by traditional ensemble methods is not sufficiently reliable.

Existing ensemble methods are machine learning-based methods and can be broadly classified into bagging and boosting. Bagging can reduce error variance and boosting can reduce error bias. Additionally, the voting strategy that determines the final result in bagging and boosting may include hard voting and soft voting. Here, in hard voting (or majority voting), all individual classifiers vote for a class, and the class voted on by a majority vote may be determined as the final output. On the other hand, in soft voting, every individual classifier can provide a probability value that a particular data point belongs to a particular target class. In this case, soft voting can calculate the average or weighted sum of the prediction probability values of all classifiers and determine the highest label value as the final prediction result.

Bagging and boosting strategies are applied to the model to be learned and can be used in various ensemble methods. For example, Random Forest is an ensemble method that compensates for the shortcomings of a single tree by using a bagging strategy for the decision tree, which is the base runner. As another example, Adaboost is a basic boosting strategy that adaptively learns by assigning higher weights to mislabeled data sets by weak learners, and finally combines the results of the models. Using similar principles, Gradient Descent Boosting weights gradients and can be trained to make the model's error smaller. Besides that, there are boosting algorithms such as XGBoost and LightGBM. Other ensemble strategies may include stacking, explicit/implicit ensemble, homogeneous/heterogeneous ensemble, and decision fusion strategies.

Deep neural networks require high computational costs for model training, and generalization performance cannot be guaranteed. However, using an ensemble strategy can improve performance by considering the diversity of each model. For example, when combining AlexNet and GoogLeNet, which are powerful CNN models for image classification, accuracy can be improved. In another example, the performance of a single CNN model can be improved by applying a boosting algorithm to the CNN. In another example, CNN AdaBoost can improve performance by repeating the process of learning the first CNN from initial data and updating the weights of the data to be learned in the next CNN. Additionally, bagging-based ensembles can improve performance by averaging the results of multiple CNN submodels.

Ensembles can also be used in tasks that use RNN-based models such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). Additionally, the ensemble can predict time series data (e.g., fluctuations and closing prices of cryptocurrency). For example, the Deep 4-LSTM technique learns each of four LSTM models and then calculates the weighted average of the resulting outputs of each LSTM model in the ensemble stage to determine the final prediction, thereby combining the single models 2-LSTM and 3 -Performance can be improved over LSTM. As another example, an ensemble method based on various time series models uses a total of six models, including RNN, LSTM, GRU, and bi-RNN, as the basic models that make up the ensemble, and the final prediction can be determined through majority voting. These ensemble methods were able to improve the performance of time series tasks by combining model results at the stage of determining the final prediction. On the other hand, Adaboost-GRU is a hybrid ensemble learning method that is a simple combination of several single models. It uses the Adaboost algorithm to obtain ensemble weights of the GRU model, which can improve performance over a single model in financial time series forecasting. You can.

Performance of sequential data can be improved using ensembles. For tasks with sequential characteristics, a sequence-to-sequence model can generally be used, and since the sequence-to-sequence model also consists of an RNN-based encoder and decoder, an ensemble effect can be expected. Ensemble techniques based on the Nematus model, which has shown excellent performance in WMT, can use a combination of a checkpoint ensemble using the last N checkpoints of a single model and an independent ensemble combining independent models trained with different hyperparameters. there is. In this case, the ensemble of Nematus models can use geometric mean and beam search decoding of the probability distributions of individual models at each time step. On the other hand, if you use the arithmetic mean and then select the voting method, you can implement the ensemble with simpler code. However, this method has the problem of ignoring the recursive structure of the RNN. Specifically, methods that ensemble the final results are not suitable for tasks using RNN-based sequence-to-sequence models that recursively influence the next output.

Figure 1 shows an example of an ensemble method of recursive neural network models (110_1 to 110_M) according to an embodiment of the present disclosure. As shown, the recursive neural network models 110_1 to 110_M may generate output for each time step 120_1 to 120_N. Additionally, each output can repeatedly affect the output of the next time step. Accordingly, the output sequence 130 corresponding to the input sequence 112 may be finally output.

In one embodiment, each of the plurality of recursive neural network models 110_1 to 110_M may receive an input sequence 112. Here, the plurality of recursive neural network models 110_1 to 110_M may be models learned with different training data. Additionally, the plurality of recursive neural network models 110_1 to 110_M may be generative models with a high degree of output diversity. Additionally or alternatively, the plurality of recursive neural network models 110_1 to 110_M may be sequence-to-sequence models.

In one embodiment, each of the recursive neural network models 110_1 to 110_M may compress the input sequence 112 from the encoder into a context vector with a fixed size and transmit it to the decoder. In this case, the decoder can generate the output sequence 130 using the context vector.

In one embodiment, outputs of a plurality of recursive neural network models (110_1 to 110_M) may be ensembled for every time step (120_1 to 120_N). Specifically, the processor may determine a settled output based on the outputs of the plurality of recursive neural network models (110_1 to 110_M) at every time step (120_1 to 120_N). Additionally, the agreed output may affect the output of the recursive neural network models 110_1 to 110_M at subsequent time steps. Through repetition of this process, the output sequence 130 corresponding to the input sequence 112 can be generated.

The ensemble method of recursive neural network models of the present disclosure can utilize structural properties of RNNs that affect predictions at the next time step after each single model is learned. Additionally, the recursive ensemble of the present disclosure can apply a soft voting algorithm and a survival algorithm according to the method of combining intermediate outputs. Here, unlike the existing ensemble method that can use heterogeneous models, a homogeneous model can be used because output must be collected at the intermediate stage of the RNN, that is, at each time step.

With this configuration, the recursive network architecture provides an output at each time step, and that output can recursively affect the next output at the next time step. Accordingly, the ensemble method of the recursive neural network model can recursively consider the output of the entire model for each time step. Additionally, because it is relatively independent of the network model, it can be universally applied to various types of recursive network models.

Figure 2 is a diagram showing the structure of a traditional majority voting ensemble model. Here, the traditional majority ensemble model may refer to an algorithm that combines only the final output of the RNN by reconstructing the existing ensemble method, which is difficult to apply to sequential data. Traditional majority vote ensemble models can combine the final results of each model without considering the results for each time step.

Specifically, the traditional majority voting ensemble model can independently learn M models (212_1 to 212_M) and then predict the result from the prediction step to the last time step in the first operation 210. Additionally, M models (212_1 to 212_M) can each generate N outputs (214_1 to 214_M) and reliability scores (216_1 to 216_M). Here, the reliability score (w ^-i ) (216_1 to 216_M) can be determined based on the average of the softmax values of the decoder output for each time step.

Each model may be an encoder-decoder sequence-to-sequence model (Encoder-Decoder Seq2Seq). For example, each model could be a GRU-based sequence-to-sequence model that uses attention for Neural Machine Translation (NMT). In another example, each model may be an LSTM based sequence-to-sequence model for string arithmetic.

In one embodiment, the GRU-based sequence-to-sequence model and the LSTM-based sequence-to-sequence model can generate a context vector containing information about the input sequence after it passes the encoder. The context vector is passed to the decoder, and the decoder can predict the next time step based on the results of the previous time step.

In the GRU-based sequence-to-sequence model, attention based on the encoder-decoder structure model can be used. The encoder and decoder can each process sequences using GRU. In addition, GRU is an RNN modified structural model that solves the long-term dependency problem of RNN, and may include a structure consisting of a reset gate and an update gate. Additionally, in the LSTM-based sequence-to-sequence model for string arithmetic, LSTM is a RNN variant structure that emerged to overcome the problem of RNNs that tend to not learn well when the distance between inputs increases, and is a memory cell ( Long-term dependency can be maintained through memory cells.

In one embodiment, the final result may be determined using a majority vote based on the confidence score among the N results. Specifically, in the second operation 220, a group having the same output may be created among the models 212_1 to 212_M participating in the ensemble. Additionally, the sum of the reliability scores of models within a group may be set as the group reliability score (W ⁱ ). That is, not only the number of models but also the reliability of the models can be considered.

The third operation 230 is an example in which the final result is determined. Among multiple groups, the output of the group with the largest sum of reliability scores can be adopted as the final result of the traditional majority vote ensemble model. In other words, if the sum of the reliability scores of the models in the group is high, it can be adopted as the final result even if the number of models in the group is small.

The pseudocode of this traditional majority vote ensemble algorithm can be expressed as Table 1 below.

maximum length source text
G, G_conf, Result

{}
for t = 1 to N do
ConfidenceScore

{c₁, c₂, · · ·, c_M}
for Model_i

S do
Generate Prediction at time-step t
Add softmax value to c_i
end for
end for
Average(ConfidenceScore)
G

Group(S) with same answer
G_conf

Sum(ConfidenceScore) for each group in G
Result

Get Group answer using max(G_conf)
Return Result Input : Trained Independently Models S = {Model _1:M }
N

maximum length source text
G, G_conf, Result

{}
for t = 1 to N do
ConfidenceScore

{c ₁ , c ₂ , · · ·, c _M }
for Model _i

S do
Generate Prediction at time-step t
Add softmax value to c _i
end for
end for
Average (ConfidenceScore)
G

Group(S) with same answer
G_conf

Sum(ConfidenceScore) for each group in G
Result

Get Group answer using max(G_conf)
Return Result

Figure 3 is a diagram showing an example process of a traditional majority voting ensemble model. The process shown represents an example of translating the Spanish "hoy es feliz domingo" into English. Five single models (Model 1 to Model 5) can translate the Spanish "hoy es feliz domingo" into an English word at each time step and obtain the softmax value for each word. Here, the correct sentence 310 is “Today is happy Sunday.”

Traditional majority vote ensemble models can determine the final result based on the group's confidence score. For example, when grouping models with the same output, model 1 is grouped into the first group (322), model 2 and model 3 are grouped into the second group (324), and model 4 and model 5 are grouped into the third group (326). It can be set to . In this case, the reliability score (W ² ) of the second group 324 is 1.915, which is the highest among the three groups, so the output sequence 330 of the traditional majority voting ensemble model can be determined as “Tomorrow is sad day.” In this case, the output sequence 330 may be different from the correct sentence 310.

Figure 4 is a diagram showing the structure of a survival ensemble model according to an embodiment of the present disclosure. Just as in a game consisting of stages, only the winner who survives can participate in the next stage, in a survival ensemble model, only the winner model that survives the prediction in the current time step can participate in the next time step.

In one embodiment, each of the M models 410_1 to 410_M may be trained independently on different data sets. Afterwards, all trained models (410_1 to 410_M) can participate in survival, starting with the decoder's start token. Each model can participate in the next time step only if it produces the correct answer in the prediction step.

In one embodiment, the standard answer that determines whether each model survives and participates in the time step may be the settled output. Here, the agreed output can be determined by a majority vote of the results output for each model at each time step. At this time, majority vote may mean that the number of output results from each model is the majority vote. Alternatively, if the majority of the output results are the same and a consensus output cannot be determined by majority vote, the result with the highest reliability value among the majority results may be determined as the consensus output. Additionally, the consensus output in machine translation may be a word selected by majority vote.

In one embodiment, losing models with outputs that differ from the agreed upon outputs may be eliminated. Here, since the eliminated loser model cannot proceed to the next time step, incorrect models that affect the final result can be filtered out in advance. By repeating this process, the output of the winner model that survived until the last time step can be determined as the output sequence of the survival ensemble model. In this case, the final winning model may be multiple.

The pseudocode of this survival ensemble algorithm can be expressed as Table 2 below.

maximum length source text
Result

{}
for t = 1 to N do
H, Winner, Loser

{}
for all Model_i

S do
Generate Prediction at time-step t
Add model prediction to H
end for
end for
SettledOutput

Majority Voting(H)
Winner

Models with the Settled Output
Loser

Models with no Settled Output
Update S

Winner
Feed SettledOutput to S
Result

Result + SettledOutput
Return Result Input : Trained Independently Models S = {Model _1:M }
N

maximum length source text
Result

{}
for t = 1 to N do
H, Winner, Loser

{}
for all Model _i

S do
Generate Prediction at time-step t
Add model prediction to H
end for
end for
SettledOutput

Majority Voting(H)
Winner

Models with the Settled Output
Loser

Models with no Settled Output
Update S

Winner
Feed SettledOutput to S
Result

Result + SettledOutput
Return Result

Figure 5 is a diagram showing an example process of a survival ensemble model according to an embodiment of the present disclosure. Like Figure 3, the process in Figure 5 represents an example of translating the Spanish "hoy es feliz domingo" into English. In a survival ensemble model, the individual output of a specific time step can be determined according to the agreed-upon output of that time step. For example, the number of models that output “Today” at the first time step (t ₁ ) (3; Model 1, Model 4, and Model 5) is the number of models that output “Tomorrow” (2; Model 5). 2 and model 3). Accordingly, “Today” output by Model 1, Model 4, and Model 5 may be determined as the agreed upon output by majority vote. In this case, Model 2 and Model 3 are loser models and cannot participate in the second time step. That is, Model 2 and Model 3 may be removed after completion of the first time step (t ₁ ) and not used in the second time step. In the second time step (t ₂ ) and the third time step (t ₃ ), " “is” and “happy” can each be determined as the agreed output. In this case, since Model 1, Model 4, and Model 5 all output “is” and “happy,” Model 1, Model 4, and Model 5 are the winning models and can continue to be used in the next time step.

Additionally, at the fourth time step (t ₄ ), model 1, model 4, and model 5 can output “Sunday”, “Monday”, and “Monday”, respectively. In this case, “Monday” may be determined as the agreed output by majority vote. As a result, the survival output sequence 530 can be determined as “Today is happy Monday” by connecting the outputs at each time step (t ₁ to t ₄ ) of the final winner model 520. This output sequence 530 may be the same or similar to the correct sentence 510.

With this configuration, incorrect answer models that affect the final result can be filtered out in advance through the survival ensemble model. Accordingly, the prediction performance of the model can be improved by reflecting the structural properties of the RNN.

FIG. 6 is a diagram illustrating an example process of a consensus ensemble model according to an embodiment of the present disclosure. Each single model produces multiple outputs at each time step, and each output may have a reliability value. In this case, a consensus output can be determined based on the reliability value of the output of each single model.

In one embodiment, each of the M models 610_1 to 610_M may be independently trained models on different data sets. Afterwards, every single model 610_1 to 610_M can be ensembled in the prediction step. Specifically, starting from the start token of the decoder, a consensus output can be determined through soft voting for the results output by all models (610_1 to 610_M) at every time step (620_1 to 620_N). Here, soft voting may mean a method of proceeding as an ensemble by considering the output of all models at each time step. Additionally, the agreed upon output can be passed as the input of every single model 610_1 to 610_M in the next time step. That is, the consensus ensemble model can feed back the intermediate output as the input of the next time step of the single model.

The pseudocode of this consensus ensemble algorithm can be expressed as Table 3 below.

maximum length source text
Result

{}
for t = 1 to N do
H

{}
for Model_i

S do
Generate Prediction at time-step t
Add Predictions to H
end for
end for
SV_t

SoftVoting(H)
Feed SV_t to S
Result

Result + SV_t
Return Result Input : Trained Independently Models S = {Model _1:M }
N

maximum length source text
Result

{}
for t = 1 to N do
H

{}
for Model _i

S do
Generate Prediction at time-step t
Add Predictions to H
end for
end for
SV _t

SoftVoting(H)
Feed SV _t to S
Result

Result + SV _t
Return Result

Figure 7 is a diagram illustrating an example process of a consensus ensemble model according to an embodiment of the present disclosure. Like Figure 3, the process in Figure 7 shows an example of translating the Spanish "hoy es feliz domingo" into English. In consensus ensemble models, soft voting can be used to pass the consensus output at each time step to the next time step. For example, at the third time step (t ₃ ), the sum of the reliability of “happy” output by the five models (Model 1 to Model 5) is greater than the sum of the reliability of “sad”, so “happy” is included in soft voting. It can be determined by the output agreed upon. Then, the agreed output “happy” at the third time step (t ₃ ) can be input to all models (models 1 to 5) at the fourth time step (t ₄ ). In this case, even though Models 2 and 3 output “sad” at the third time step (t ₃ ), “happy”, which is the agreed output, can be input to Models 2 and 3 at the fourth time step (t ₄ ). there is.

In one embodiment, each model is shown as generating one output at each time step, but the present invention is not limited thereto, and each model may generate a plurality of outputs at each time step. For example, each model may produce n high-confidence outputs (e.g., n=3) at each time step. For example, at the fourth time step (t ₄ ), Model 1 outputs softmax values for words such as [Sunday:0.99, day:0.7, Monday:0.1], and Model 2 outputs softmax values for words such as [Sunday:0.89, day. :0.9, Monday:0.7], Model 3 outputs [Sunday:0.88, day:0.96, Monday:0.67], Model 4 outputs [Sunday:0.55, day:0.45, Monday:0.95], and Model 5 can output [Sunday:0.82, day:0.34, Monday:0.85]. In this case, by averaging the softmax values for each word, “Sunday” can be determined as the agreed output at the fourth time step (t ₄ ) with the highest probability of 0.81.

In one embodiment, the output sequence 720 can be determined by concatenating the agreed upon outputs at all time steps. Accordingly, the output sequence of the consensus ensemble model in Figure 7 may be determined as “Today is happy Sunday.” This output sequence 720 may be the same or similar to the correct sentence 710.

Through this configuration, the agreed output can be fed back to the loser model without eliminating the loser model through the consensus ensemble model. Accordingly, the performance of the ensemble model can be improved because all models participate at every time step.

FIG. 8 is a diagram illustrating graphs 810 and 820 comparing the performance of the ensemble model of the present disclosure, a traditional majority voting ensemble model, and a single model. The first graph 810 shows the results of comparing the translation performance of the consensus ensemble model, survival ensemble model, and traditional majority vote ensemble model with the single model used in the ensemble. The second graph 820 shows the results of comparing the string arithmetic accuracy of the single model used in the ensemble of the consensus ensemble model, survival ensemble model, and traditional majority vote ensemble model.

According to the first graph 810, in Spanish-English neural network machine translation, the traditional majority ensemble model, survival ensemble model, and consensus ensemble model have better performance than a single model. Here, the neural network machine translation performance evaluation uses Python TQE (Package Translation Quality Estimation), and the higher the TQE, the higher the quality of the translation.

In addition, according to the first graph 810, the survival ensemble model has the effect of removing noise from the final output sequence by determining the agreed output at each time step, resulting in higher TQE performance than the traditional majority voting ensemble model. Additionally, the consensus ensemble model has higher TQE performance than the survival ensemble model by not removing a single model that makes an incorrect prediction but feeding back the single model to participate in the next time step.

Table 4 below shows the TQE values for the average of the three ensemble models and 15 single models used for Spanish-English machine translation. Among the three ensemble models, the survival ensemble model and consensus ensemble model have higher TQE scores than the traditional majority ensemble model by considering the intermediate output of each time step. In other words, the recursive ensemble method that reflects the structural properties of RNN is confirmed to be more effective than the method of merging the final output using the existing ensemble concept.

Type Model TQE ensemble model agreement 0.743250 survival 0.731033 traditional majority vote 0.724257 single model Single (Avg) 0.690916

The example in Table 5 below shows the result of translating the Spanish "Me temo que no hay mucho que pueda hacer para ayudar" into English. In this case, it is confirmed that the consensus ensemble model achieved the highest TQE score and produced translation results similar to the correct answer. Due to differences in the characteristics of Spanish and English, the words output at each time step may differ in word order from the correct answer, but the context of the overall sentence can be maintained. Additionally, the survival ensemble model outputs words that are similar to the correct answer, most matching the correct answer at each time step. However, because it outputs the word “left”, which is unrelated to the correct answer, the TQE value is lower than the consensus ensemble model. Comparatively, the traditional majority vote ensemble model fails to capture some important words compared to the recursive ensemble method, resulting in poor machine translation quality.

Ground Truth Spanish: Me temo que no hay mucho que pueda hacer para ayudar.
English: There is not much I can do to help, I am afraid. Method Hypothesis TQE agreement i m afraid there s not much for me to do to help. 0.957 survival i m afraid there s not much left that i can do to help. 0.952 traditional majority vote i m afraid there s not long to do to do. 0.721

According to the second graph 820, in string arithmetic, the traditional majority ensemble model, survival ensemble model, and consensus ensemble model have better accuracy than the single model. The traditional majority voting ensemble model is confirmed to perform better than the single model, achieving 64.70% accuracy with 76.4% accuracy. In addition, the survival ensemble model and the consensus ensemble model are 77.5% and 86.8%, respectively, confirming superior performance over the traditional majority ensemble model. In particular, the consensus ensemble model is confirmed to have an accuracy of 78% after learning for only 100 epochs.

According to these results, it is confirmed that the ensemble model of the present disclosure has higher performance by connecting the intermediate prediction steps of each single model participating in the ensemble.

9 is a flow chart illustrating an example method 900 according to one embodiment of the present disclosure. In one embodiment, method 900 may be performed by a user terminal or at least one processor of an information processing system. Method 900 may begin with a processor receiving an input sequence (S910).

Afterwards, the processor may receive a plurality of recursive neural network models to process the input sequence (S920). Here, a plurality of recursive neural network models may each be trained with different learning data. Additionally, the plurality of recursive neural network models may be generative models. Additionally, the plurality of recursive neural network models may be sequence-to-sequence models.

Afterwards, the processor may generate an output sequence corresponding to the input sequence by ensembleing the outputs of a plurality of recursive neural network models at each time step (S930). Additionally, the processor may output an output sequence (S940).

In one embodiment, the processor may generate output of a plurality of recursive neural network models in a first time step. Additionally, the processor may determine a settled output based on the outputs of the plurality of recursive neural network models in the first time step. Here, the agreed output may be determined based on the number of models with the same output among a plurality of recursive neural network models in the first time step.

In one embodiment, the processor may eliminate recursive neural network models that have outputs that differ from the agreed upon output. In this case, a recursive neural network model that is not removed at each time step can use the output it generated in the previous time step as the input of the current time step. Additionally, the processor may generate an output sequence corresponding to the input sequence by connecting the output at each time step of at least one recursive neural network model remaining to the last time step.

In one embodiment, the processor may generate outputs of a plurality of recursive neural network models in a first time step. Additionally, the processor may determine a settled output based on the outputs of the plurality of recursive neural network models in the first time step. Here, the agreed output may be determined based on the confidence value of the output of the plurality of recursive neural network models in the first time step.

In one embodiment, the processor may generate outputs of a plurality of recursive neural network models by using the output agreed upon in the second time step as input to all of the plurality of recursive neural network models. Additionally, the processor can concatenate the agreed upon outputs at all time steps to generate an output sequence corresponding to the input sequence.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. Media may be used to continuously store executable programs on a computer, or may be temporarily stored for execution or download. Additionally, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and There may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The methods, operations or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

When implemented in software, the techniques may be stored on or transmitted through a computer-readable medium as one or more instructions or code. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or the desired program code in the form of instructions or data structures. It can be used to transfer or store data and can include any other media that can be accessed by a computer. Any connection is also properly termed a computer-readable medium.

For example, if the Software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, , fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. As used herein, disk and disk include CD, laser disk, optical disk, digital versatile disc (DVD), floppy disk, and Blu-ray disk, where disks are usually magnetic. It reproduces data optically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known. An exemplary storage medium may be coupled to the processor such that the processor may read information from or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage media may reside within an ASIC. ASIC may exist within the user terminal. Alternatively, the processor and storage medium may exist as separate components in the user terminal.

Figure 10 is a schematic diagram showing a configuration in which an information processing system 1030 is connected to communicate with a plurality of user terminals 1010_1, 1010_2, and 1010_3 in order to generate an output sequence according to an embodiment of the present disclosure. As shown, a plurality of user terminals 1010_1, 1010_2, and 1010_3 may be connected to an information processing system 1030 that can provide an output sequence generation service through a network 1020. Here, the plurality of user terminals 1010_1, 1010_2, and 1010_3 may include terminals of users provided with an output sequence generation service.

In one embodiment, information processing system 1030 may be one or more server devices and/or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data associated with providing output sequence generation services, etc. , or may include one or more distributed computing devices and/or distributed databases based on cloud computing services.

The output sequence generation service provided by the information processing system 1030 may be provided to users through an output sequence generation service application web browser or web browser extension program installed on each of the plurality of user terminals 1010_1, 1010_2, and 1010_3. You can. For example, the information processing system 1030 may provide information corresponding to an output sequence creation request received from the user terminals 1010_1, 1010_2, and 1010_3 through an output sequence creation service application or perform corresponding processing. .

A plurality of user terminals 1010_1, 1010_2, and 1010_3 may communicate with the information processing system 1030 through the network 1020. The network 1020 may be configured to enable communication between a plurality of user terminals 1010_1, 1010_2, and 1010_3 and the information processing system 1030. Depending on the installation environment, the network 1020 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of wireless networks such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and includes communication methods that utilize communication networks that the network 1020 may include (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.), as well as user terminals (1010_1, 1010_2, 1010_3) ) may also include short-range wireless communication between

In FIG. 10, the mobile phone terminal (1010_1), tablet terminal (1010_2), and PC terminal (1010_3) are shown as examples of user terminals, but they are not limited thereto, and the user terminals (1010_1, 1010_2, 1010_3) use wired and/or wireless communication. This is possible and may be any computing device on which an output sequence generation service application or a web browser, etc. can be installed and executed. For example, user terminals include AI speakers, smartphones, mobile phones, navigation, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, It may include wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, set-top boxes, etc. In addition, in Figure 10, three user terminals (1010_1, 1010_2, 1010_3) are shown communicating with the information processing system 1030 through the network 1020, but this is not limited to this, and a different number of user terminals are connected to the network ( It may also be configured to communicate with the information processing system 1030 via 1020).

Figure 11 is a block diagram showing the internal configuration of the user terminal 1010 and the information processing system 1030 according to an embodiment of the present disclosure. The user terminal 1010 may refer to any computing device capable of executing applications, web browsers, etc. and capable of wired/wireless communication, for example, the mobile phone terminal 1010_1, tablet terminal 1010_2 of FIG. 10, It may include a PC terminal (1010_3), etc. As shown, the user terminal 1010 may include a memory 1112, a processor 1114, a communication module 1116, and an input/output interface 1118. Similarly, information processing system 1030 may include memory 1132, processor 1134, communication module 1136, and input/output interface 1138. As shown in FIG. 11, the user terminal 1010 and the information processing system 1030 are configured to communicate information and/or data through the network 1020 using respective communication modules 1116 and 1136. It can be. Additionally, the input/output device 1120 may be configured to input information and/or data to the user terminal 1010 through the input/output interface 1118 or to output information and/or data generated from the user terminal 1010.

Memories 1112 and 1132 may include any non-transitory computer-readable recording medium. According to one embodiment, the memories 1112 and 1132 are non-permanent mass storage devices such as read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It can be included. As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the user terminal 1010 or the information processing system 1030 as a separate persistent storage device that is distinct from memory. Additionally, an operating system and at least one program code may be stored in the memories 1112 and 1132.

These software components may be loaded from a computer-readable recording medium separate from the memories 1112 and 1132. This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal 1010 and the information processing system 1030, for example, a floppy drive, disk, tape, DVD/CD- It may include computer-readable recording media such as ROM drives and memory cards. As another example, software components may be loaded into the memories 1112 and 1132 through communication modules 1116 and 1136 rather than computer-readable recording media. For example, at least one program is loaded into memory 1112, 1132 based on a computer program installed by files provided over the network 1020 by developers or a file distribution system that distributes installation files for applications. It can be.

The processors 1114 and 1134 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to processors 1114 and 1134 by memories 1112 and 1132 or communication modules 1116 and 1136. For example, processors 1114 and 1134 may be configured to execute received instructions according to program codes stored in recording devices such as memories 1112 and 1132.

The communication modules 1116 and 1136 may provide a configuration or function for the user terminal 1010 and the information processing system 1030 to communicate with each other through the network 1020, and may provide a configuration or function for the user terminal 1010 and/or information processing. The system 1030 may provide a configuration or function for communicating with other user terminals or other systems (for example, a separate cloud system, etc.). For example, a request or data (e.g., a request to generate an output sequence, etc.) generated by the processor 1114 of the user terminal 1010 according to a program code stored in a recording device such as the memory 1112 is sent to the communication module 1116. It may be transmitted to the information processing system 1030 through the network 1020 under the control of . Conversely, a control signal or command provided under the control of the processor 1134 of the information processing system 1030 passes through the communication module 1136 and the network 1020 and then through the communication module 1116 of the user terminal 1010. It may be received by the user terminal 1010.

The input/output interface 1118 may be a means for interfacing with the input/output device 1120. As an example, input devices may include devices such as cameras, keyboards, microphones, mice, etc., including audio sensors and/or image sensors, and output devices may include devices such as displays, speakers, haptic feedback devices, etc. You can. As another example, the input/output interface 1118 may be a means for interfacing with a device that has a structure or function for performing input and output, such as a touch screen, integrated into one. For example, the processor 1114 of the user terminal 1010 uses information and/or data provided by the information processing system 1030 or another user terminal when processing instructions of a computer program loaded in the memory 1112. A service screen, etc. constructed as such may be displayed on the display through the input/output interface 1118. In FIG. 11 , the input/output device 1120 is shown not to be included in the user terminal 1010, but the present invention is not limited to this and may be configured as a single device with the user terminal 1010. Additionally, the input/output interface 1138 of the information processing system 1030 may be connected to the information processing system 1030 or means for interfacing with a device (not shown) for input or output that the information processing system 1030 may include. It can be. In FIG. 11 , the input/output interfaces 1118 and 1138 are shown as elements configured separately from the processors 1114 and 1134, but the present invention is not limited thereto, and the input/output interfaces 1118 and 1138 may be configured to be included in the processors 1114 and 1134. there is.

The user terminal 1010 and the information processing system 1030 may include more components than those in FIG. 11 . However, there is no need to clearly show most prior art components. In one embodiment, the user terminal 1010 may be implemented to include at least some of the input/output devices 1120 described above. Additionally, the user terminal 1010 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, if the user terminal 1010 is a smartphone, it may include components generally included in a smartphone, such as an acceleration sensor, a gyro sensor, a microphone module, a camera module, and various physical devices. Various components such as buttons, buttons using a touch panel, input/output ports, and vibrators for vibration may be implemented to be further included in the user terminal 1010.

While a program for an output sequence generation service application, etc. is operated, the processor 1114 inputs input through an input device such as a touch screen, keyboard, camera including an audio sensor and/or an image sensor, and a microphone connected to the input/output interface 1118. or receive selected text, images, videos, voices, and/or actions, and store the received text, images, videos, voices, and/or actions in the memory 1112 or use the communication module 1116 and the network ( It can be provided to the information processing system 1030 through 1020).

The processor 1114 of the user terminal 1010 manages, processes, and/or stores information and/or data received from the input/output device 1120, other user terminals, the information processing system 1030, and/or a plurality of external systems. It can be configured to do so. Information and/or data processed by processor 1114 may be provided to information processing system 1030 via communication module 1116 and network 1020. The processor 1114 of the user terminal 1010 may transmit and output information and/or data to the input/output device 1120 through the input/output interface 1118. For example, the processor 1114 may display the received information and/or data on the screen of the user terminal 1010.

The processor 1134 of the information processing system 1030 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals 1010 and/or a plurality of external systems. Information and/or data processed by the processor 1134 may be provided to the user terminal 1010 through the communication module 1136 and the network 1020.

10 and 11, the user terminal transmits an input sequence to the information processing system through a network, the information processing system generates an output sequence based on the input sequence, and the information processing system transmits the generated output sequence to the user terminal. Although shown, it is not limited thereto. For example, the user terminal may generate an output sequence on its own based on the input sequence, without communicating with the information processing system.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by a person skilled in the art to which the invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

110: model
112: Input sequence
120: Time step
130: Output sequence

Claims (13)

In the ensemble method of a recurrent neural network model, performed by at least one processor,
receiving an input sequence;
Receiving a plurality of recursive neural network models to process the input sequence;
generating an output sequence corresponding to the input sequence by ensembleing outputs of the plurality of recursive neural network models at every time step; and
Outputting the output sequence
Ensemble method of recursive neural network model, including.
According to paragraph 1,
An ensemble method of a recursive neural network model, wherein the plurality of recursive neural network models are each learned with different learning data.
According to paragraph 1,
An ensemble method of a recursive neural network model, wherein the plurality of recursive neural network models are generative models.
According to paragraph 1,
An ensemble method of a recursive neural network model, wherein the plurality of recursive neural network models are sequence-to-sequence models.
According to paragraph 1,
The step of generating the output sequence is,
generating outputs of the plurality of recursive neural network models at a first time step;
determining a settled output based on outputs of the plurality of recursive neural network models at the first time step; and
Removing recursive neural network models with outputs different from the agreed output
Ensemble method of recursive neural network model, including.
According to clause 5,
The ensemble method of a recursive neural network model, wherein the agreed output is determined based on the number of models having the same output among the plurality of recursive neural network models in the first time step.
According to clause 5,
An ensemble method of recursive neural network models in which a recursive neural network model that is not removed at each time step uses the output it generated in the previous time step as the input of the current time step.
According to clause 5,
The step of generating the output sequence is,
Generating an output sequence corresponding to the input sequence by connecting the output at each time step of at least one recursive neural network model remaining at the last time step.
An ensemble method of a recursive neural network model, further comprising:
According to paragraph 1,
The step of generating the output sequence is,
generating outputs of the plurality of recursive neural network models at a first time step;
determining an agreed output based on the outputs of the plurality of recursive neural network models at the first time step; and
Using the agreed output as an input to all of the plurality of recursive neural network models in a second time step, generating outputs of the plurality of recursive neural network models.
Ensemble method of recursive neural network model, including.
According to clause 9,
The ensemble method of a recursive neural network model, wherein the agreed output is determined based on reliability values of the outputs of the plurality of recursive neural network models at the first time step.
According to clause 9,
The step of generating the output sequence is,
Connecting the agreed upon outputs at all time steps to generate an output sequence corresponding to the input sequence.
An ensemble method of a recursive neural network model, further comprising:
A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1 to 11 on a computer.
As an information processing system,
communication module;
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory,
The at least one program is,
receives an input sequence,
Receiving a plurality of recursive neural network models to process the input sequence,
Ensemble outputs of the plurality of recursive neural network models at each time step to generate an output sequence corresponding to the input sequence,
An information processing system comprising instructions for outputting the output sequence.

Images