KR102334268B1 - A system for predicting user drop out rate based on artificial intelligence learning and method thereof - Google Patents

Abstract

The present invention relates to a user knowledge tracking method with improved accuracy. In the method of operating a user churn rate prediction system including a plurality of encoder neural networks and a plurality of decoder neural networks, problem information is input to the kth encoder neural network, and , inputting response information into the kth decoder neural network, generating query data that is information about a problem for which a user wants to predict the probability of correct answer by reflecting a weight in the response information, and reflecting the weight in the problem information to make the query generating attention information to be used as a weight for data; and learning the user churn rate prediction system by using the attention information as a weight for the query data.

Description

A SYSTEM FOR PREDICTING USER DROP OUT RATE BASED ON ARTIFICIAL INTELLIGENCE LEARNING AND METHOD THEREOF

The present invention relates to a system for predicting a user's churn rate based on artificial intelligence learning and an operating method thereof. Specifically, it is an invention related to a user churn rate prediction system for predicting a user churn rate by inputting problem information to an encoder neural network of a transformer structure and response information to a decoder neural network.

Recently, the use of the Internet and electronic devices has been actively carried out in each field, and the educational environment is also changing rapidly. In particular, with the development of various educational media, learners can choose and use a wider range of learning methods. Among them, the education service through the Internet has been positioned as a major teaching and learning method because of the advantage of overcoming time and spatial constraints and enabling low-cost education.

By combining this online education service with artificial intelligence of various models, it is possible to provide more efficient learning contents by predicting the probability of a user's correct answer to a random problem that was previously impossible in an offline education environment. However, since users in the mobile environment are more vulnerable to various factors that cause departure from the online learning environment during learning, the departure from the online environment may occur more frequently than the departure from the offline environment. For example, whenever a call comes in, a text message is received, a SNS notification such as Facebook or Instagram is received, or other various notifications are received, the user leaves the learning program to check these information. , which in turn leads to a decrease in learning efficiency. Methods to prevent user churn in the offline environment have been continuously studied, but research on churn during learning in the online environment has not been conducted yet.

Recently, among various artificial intelligence models, the transformer model does not use RNN and uses only attention to create an encoder-decoder structure. It has the advantage of fast learning speed and superior performance than RNN, so it has been applied to the field of online education.

Using the transformer model, it is possible to predict the probability that the user will deviate from the online learning environment. How to configure the data input to the encoder and decoder of the transformer model to obtain inference results optimized for predicting the user churn rate in the online learning environment A method for more effectively predicting the churn rate using the transformer model is required, such as whether it can be possible or what method can be used to prevent the churn rate from being predicted based on the problem that the user has not yet solved due to the nature of the learning content.

The present invention is an invention to solve the above-mentioned problem, and proposes a transformer structure-based artificial intelligence model and input data format optimized for churn rate prediction. It has the effect of predicting with high accuracy.

The present invention relates to a user knowledge tracking method with improved accuracy. In the method of operating a user churn rate prediction system including a plurality of encoder neural networks and a plurality of decoder neural networks, problem information is input to the kth encoder neural network, and , inputting response information into the kth decoder neural network, generating query data that is information about a problem for which a user wants to predict the probability of correct answer by reflecting a weight in the response information, and reflecting the weight in the problem information to make the query generating attention information to be used as a weight for data; and learning the user churn rate prediction system by using the attention information as a weight for the query data.

An artificial intelligence learning-based user churn rate prediction system and an operating method thereof according to an embodiment of the present invention propose a transformer structure-based artificial intelligence model and input data format optimized for churn rate prediction, and through this, There is an effect of predicting the bounce rate of users learning in a mobile environment with higher accuracy.

1 is a diagram for explaining a user churn rate prediction system according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining in detail the operation of the user churn rate prediction system of FIG. 1 .
3 is a view for explaining drop out according to an embodiment of the present invention.
4 is a diagram for explaining key-query masking and upper triangular masking.
5 is a diagram for explaining the configuration of input data.
6 is a table for explaining the AUC of the user churn rate prediction system according to an embodiment of the present invention.
7 is a graph for explaining AUC of a user churn rate prediction system according to a sequence size of input data.
8 is a graph for explaining the AUC of the user churn rate prediction system according to the format of the input data.
9 is a flowchart illustrating an operation of a system for predicting a user churn rate according to an embodiment of the present invention.
10 is a flowchart for explaining in detail the operation of the problem information processing unit or the response information processing unit according to an embodiment of the present invention.

Specific structural or step-by-step descriptions for the embodiments according to the concept of the present invention disclosed in this specification or application are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and implementation according to the concept of the present invention Examples may be implemented in various forms, and embodiments according to the concept of the present invention may be implemented in various forms and should not be construed as being limited to the embodiments described in the present specification or application.

Since the embodiment according to the concept of the present invention may have various changes and may have various forms, specific embodiments will be illustrated in the drawings and described in detail in the present specification or application. However, this is not intended to limit the embodiment according to the concept of the present invention with respect to a specific disclosed form, and should be understood to include all changes, equivalents or substitutes included in the spirit and scope of the present invention.

Terms such as first and/or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one element from another element, for example, without departing from the scope of the present invention, a first element may be called a second element, and similarly The second component may also be referred to as the first component.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle. Other expressions describing the relationship between elements, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", should be interpreted similarly.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate that the stated feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers. , it is to be understood that it does not preclude the possibility of the existence or addition of steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

In describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to more clearly convey the gist of the present invention without obscuring the gist of the present invention by omitting unnecessary description.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a user churn rate prediction system according to an embodiment of the present invention.

Referring to FIG. 1 , the user churn rate prediction system 5 may include embedding performers 10 and 30 , an encoder neural network 20 , and a decoder neural network 40 .

The Transformer model follows the encoder-decoder, which is the structure of the existing seq2seq, and is implemented only with attention. Although the transformer model does not use RNN, it maintains an encoder-decoder structure that receives an input sequence from an encoder and outputs an output sequence from a decoder like the existing seq2seq, and has the characteristics that N units of encoder and decoder can exist.

The attention mechanism was proposed to solve the problem of vanishing gradient and information loss that comes from compressing all information into one fixed-size vector, which has been pointed out as a problem of seq2seq based on RNN.

According to the attention mechanism, at every time step that the decoder predicts the output word, the entire input data from the encoder is consulted once again. However, rather than referencing all the input data at the same rate, attention is focused on the part of the input data that is related to the data to be predicted at that time.

Referring back to FIG. 1 , the user churn rate prediction system 5 according to an embodiment of the present invention may predict the probability (the churn rate) of whether a user who is learning in a mobile environment will terminate the learning program. The information on the bounce rate may be bounce rate information.

Bounce rate information may be generated based on time. Bounce rate information may be updated with a different value every hour during learning.

Furthermore, the dropout rate information may be generated based on the user's learning characteristics, such as the correct answer rate of the problems solved so far, the average problem solving time, and the usual learning time, as well as the characteristics of the problem itself, such as the difficulty of the problem.

Since users in the mobile environment are more vulnerable to various factors that cause churn, churn in the online environment may occur more frequently than churn in the offline environment.

For example, whenever a call comes in, a text message is received, a SNS notification such as Facebook or Instagram is received, or other various notifications are received, the user leaves the learning program to check these information. , which may eventually lead to a decrease in learning efficiency.

In the past, various methods have been devised to prevent departure from offline learning and increase student concentration. However, these methods are limited to the offline environment and are difficult to apply in the online environment. There was a need for a method to predict the dropout rate and increase concentration.

The user churn rate prediction system 5 according to an embodiment of the present invention predicts the churn rate based on the transformer structure optimized for the online learning environment, and uses the format optimized for predicting the churn rate for input data. You can predict the churn rate with accuracy.

Referring back to FIG. 1 , the embedding performing units 10 and 30 may perform embedding on input data. In an embodiment of the present invention, the input data may include problem information and response information.

The problem information may be information on problems having various types and difficulties provided to measure the user's knowledge level. The problem information may include metadata of the problem information and/or the relative position of the problem in each session. A session may be a learning unit divided according to time based on the time when the user leaves.

The response information may be information on an answer selected by the user in response to the problem information. The response information may include not only the user's correct answer to the problem itself, but also information about the time taken to solve the problem given to the user.

According to an embodiment, such problem information and response information may constitute a set of problem-response log data. The problem information and response information corresponding thereto may be matched as problem-response log data, and may be input to the user churn rate prediction system 5 as a single data unit.

In an embodiment, the problem information may be expressed as 'E' as an abbreviation for Example, the response information may be expressed as 'R' as an abbreviation of Response, and the departure rate information may be expressed as 'd' as an abbreviation for Drop out. Problem-response log data can be expressed as 'I' as an abbreviation for Interaction.

The embedding performing units 10 and 30 may express input data, for example, a problem, a response, or a set of problems and responses as a vector, and perform a function of embedding it in the user knowledge tracking system 5 . There may be various ways to express the input data as a vector for the latent space. For example, it may be one of the ways to quantify and use words within artificial intelligence. Even if the expression or form input by the user is different, the meaning of words, sentences, and texts can be written while calculating the correlation and expressing this through numerical values.

As will be described later in the description of FIG. 5 , the problem information may be calculated by adding up embeddings of at least any one of problem identification information, category information, start time information, input sequence location information, and session location information. The response information may be calculated by adding up embeddings of at least one of start time information, input sequence location information, session location information, response accuracy information, required time information, in-time solution information, and bounce rate information.

The problem information expressed as a vector may be embedded in the embedding performing unit 10 and input to the encoder neural network 20 . Response information expressed as a vector may be embedded in the embedding performing unit 30 and input to the decoder neural network 40 .

The encoder neural network 20 may generate attention information based on embedded problem information. The attention information may be problem information weighted through a plurality of layers of the encoder neural network 20 . In particular, the attention information may be information generated through self-attention in the encoder neural network. Attention information may be mathematically expressed as a probability, and the sum of all attention information is 1.

Attention information may be input to the decoder neural network 40 and used as a weight for query data of the decoder neural network 40 to train the user churn rate prediction system 5 .

The artificial neural network can use attention information to learn which part is important according to the objective function. In particular, self-attention refers to performing attention on oneself, and may be an operation in which a weight is given to a part to be considered important in specific data itself, and this is reflected back to oneself. In the existing attention in seq2seq, the correlation was found with information of different data such as encoder-side data and decoder-side data.

The user churn rate prediction system 5 transmits the entire input data E1, E2, ..., Ek, R1, R2, ... , Rk-1) can be referred to again, and according to the attention information, it is possible to focus on data related to the corresponding output result.

The decoder neural network 40 may generate bounce rate information based on embedded response information and attention information. The decoder neural network 40 may perform multi-head attention for performing the aforementioned self-attention at least once on the response information.

As such, the decoder neural network 40 generates bounce rate information by performing multi-head attention on the query data generated from the response information based on the attention information weighted according to the importance in the problem information in the encoder neural network 20 . can do.

According to an embodiment of the present invention, by using the problem information for the encoder and response information for the decoder as input data optimized for predicting the user bounce rate, the system 5 for predicting the user bounce rate with improved performance can be implemented. The user churn rate prediction system 5 may train the user churn rate prediction system 5 to predict the churn rate using problem information and response information as input data. Thereafter, the user churn rate prediction system may output the churn rate information from the learned user churn rate prediction system. This is an inference process, which processes input data according to the weight determined in the learning process, and outputs information about the churn rate indicating the probability that the user will deviate while solving the problem, while listening to or watching an explanation or lecture. can

In addition, the present invention uses upper triangular masking for the encoder neural network and the decoder neural network of the transformer structure optimized for the online learning environment, thereby preventing the user from predicting the churn rate from the problem that the user has not yet solved. A bounce rate prediction system 5 may be implemented.

FIG. 2 is a diagram for explaining in detail the operation of the user churn rate prediction system of FIG. 1 .

Referring to FIG. 2 , the user churn rate prediction system 5 may include an encoder neural network 20 and a decoder neural network 40 . Again, the encoder neural network 20 includes the problem information processing unit 21 and the non-linearization performing unit 22, and the decoder neural network 40 includes the first response information processing unit 41, the second response information processing unit 42, and the non-linearization It may include a performer 43 .

Although the embedding performing unit of FIG. 1 is omitted in FIG. 2 , the embedding operation of input data may be understood with reference to the description of FIG. 1 .

The problem information may be composed of a plurality of problems (E1, E2, ..., Ek) expressed as vectors. The response information may consist of a user's responses (R1, R2, ..., Rk-1) to each of a plurality of problems (E1, E2, ..., Ek) expressed in vectors. Bounce rate information may be composed of individual bounce rate information (d1, d2, ..., dk) indicating the user's churn probability for each problem expressed as a vector.

In an embodiment, the bounce rate information dk may include R1 where the user response to the problem E1 is R1, the user response to the problem E2 is R2, . . . , when the user response to the problem Ek-1 is Rk-1, it may be information about the probability of leaving the learning program when the user is solving the problem to follow.

Although the information regarding the probability that the user will terminate the learning program during problem solving has been described as the departure rate information, according to an embodiment, the departure rate information can be used to terminate the learning program while reading the description of the problem solved by the user or watching a lecture. Probability may be included.

In this case, the departure rate information as a probability that the user will deviate from the learning program may be generated at a time exceeding the time when the problem solving is completed from the time when the problem is provided to the user. This will be described in detail in the description of FIG. 3 to be described later.

The problem information processing unit 21 may receive problem information and perform a series of operations related to self-attention. In this operation, problem information is divided into query, key, and value, a plurality of head values for each value are generated, a weight is generated from a plurality of query head values and a plurality of key head values, and the generated weight is masked. and generating prediction data by applying the masked weight to the plurality of value head values.

The prediction data generated by the problem information processing unit 21 may be attention information.

In particular, the problem information processing unit 21 may perform upper triangular masking as well as key-query masking during the masking operation. Key-query masking and upper triangular masking will be described in detail in the description of FIG. 4 to be described later.

The non-linearization performing unit 22 may perform an operation of non-linearizing the prediction data output from the problem information processing unit 21 . The ReLU function can be used for non-linearity.

Although not shown in the figure, at least one encoder neural network 20 may exist. Attention information generated in the encoder neural network 20 is again input to the encoder neural network 20, and a series of operations related to self-attention and non-linearization may be repeated several times.

Thereafter, the attention information may be divided into a key and a value value and input to the second response information processing unit. The attention information may be used as a weight for the query data input to the second response information processing unit to learn the user knowledge departure rate prediction system 5 .

The first response information processing unit 41 may receive response information and perform a series of operations related to self-attention similar to the problem information processing unit 21 . In this operation, problem information is divided into query, key, and value, a plurality of head values for each value are generated, a weight is generated from a plurality of query head values and a plurality of key head values, and the generated weight is masked. and generating prediction data by applying the masked weight to the plurality of value head values.

The prediction data generated by the first response information processing unit 41 may be query data.

The second response information processing unit 42 may receive query data from the first response information processing unit, attention information from the encoder neural network 20 , and output bounce rate information.

Attention information may be input to the decoder neural network 40 and used as a weight for query data of the decoder to train the user churn rate prediction system 5 .

The attention information may be information about a weight given to focus on a specific area of the query data. Specifically, the user churn rate prediction system 5 transmits the entire input data E1, E2, ..., Ek, R1, R2 to the encoder neural network 20 at every point in time to predict the output result d to the decoder neural network 40 . , ..., Rk-1) can be referred to again, and attention can be paid to the data related to the corresponding output result.

According to the above operation, the second response information processing unit 42 may generate dk, which is the user's departure rate information for the problem information Ek.

Although not shown in the drawing, at least one decoder neural network 40 may exist. The churn rate information generated in the decoder neural network 40 is input to the decoder neural network 40 again, and a series of operations related to self-attention, multi-head attention, and non-linearization may be repeated several times.

Like the problem information processing unit 21, the first response information processing unit 41 and the second response information processing unit perform key-query masking as well as upper triangular masking during the masking operation. can

3 is a view for explaining drop out according to an embodiment of the present invention.

Referring to FIG. 3 , the user solves a given problem (Problem 1, Problem 2, Problem 3), reads explanations (Explanation 1, Description 2, and Description 3) for each problem (Read), lectures ( You can watch lecture 1 and lecture 2) (Watch).

In FIG. 3 , the explanation and lecture for the problem 1 may be explanation 1 and lecture 1 and 2, the explanation for the problem 2 may be explanation 2, and the explanation for the problem 3 may be explanation 3. The number of explanations and lectures for each problem is not limited, and any one of the explanations and lectures may be omitted, or a plurality of explanations or lectures may be provided.

In addition, although the description is illustrated as read and the lecture is illustrated, according to an embodiment, a person's five senses in consideration of learning efficiency may be utilized in various ways.

In FIG. 3 , the time information may be the time when each activity started, the time the activity ended, the midpoint between the start and end times, or a time selected according to a preset criterion between the start and the end. . Hereinafter, the time information is assumed to be the start time of each activity and explained.

In an embodiment, the user's problem solving for problem 1 may be started at t1. After the solution of problem 1 is finished at t2, the learning program can provide explanation 1 and lecture 1. After explanation 1 is provided until t3, lecture 1 may be provided.

The user may drop out of the learning program at t4. The leaving operation may include various situations in which the user no longer uses the learning program, such as the screen of the mobile device is turned off, the learning program is switched to the background state when another application is turned on, or the power of the mobile device is turned off. .

delete

For example, the embodiment of FIG. 3 illustrates a case where the preset time is 1 hour, where the session may be defined as a series of learning activities in which the time interval between adjacent activities is less than 1 hour, and the departure is the user It can be defined as a case where the time of inactivity lasted more than 1 hour.

The preset time may be variously set other than one hour, and may be set differently for each problem according to the problem type (or category). For example, in the TOEIC test, the listening part may be set to a shorter time than the reading part, the longer the passage in the same listening part, the longer the time may be set, or in the same reading part, the shorter the passage, the shorter the time may be set. .

When the user leaves the learning program at t4, the learning program may store the state of lecture 1 being reproduced at the departure time t4. After that, when the user executes the learning program again, the automatically stored lecture may be reproduced from the time point t4 at which the user left off. The replayed lecture may be lecture 2.

According to the above-described process, Lecture 2, Question 2, Description 2, and Question 3 may be provided to the user. Since more than one hour has elapsed from t7, when problem 3 is provided, to t8, when explanation 3 is provided as the next activity, the user churn rate prediction system may determine that the user has churned.

The user churn rate prediction system can predict such user churn in advance using an artificial intelligence model and provide user-customized learning data. As described above, since the user's churn may occur even after problem solving is finished and explanations or lectures are provided, the user churn rate prediction system can predict the churn time without being limited by the user's problem solving time.

4 is a diagram for explaining key-query masking and upper triangular masking.

Although FIG. 4 shows that upper triangular masking is performed after key-query masking, both may be performed simultaneously, and upper triangular masking may be performed first.

Key-query masking may optionally be an operation that prevents attention from being performed by imposing a penalty on a value without a value (zero padding). A value of prediction data on which key-query masking is performed may be expressed as 0, and the remaining part may be expressed as 1.

In the key-query masking of FIG. 4 , for convenience of description, the last values of the query and the key are masked, but this may be variously changed according to embodiments.

Upper triangular masking may be an operation for preventing attention from being performed on information corresponding to a future location (position) for prediction of the next problem. For example, it may be an operation of masking to prevent a predicted value from being calculated from a problem that the user has not yet solved. Like the key-query masking, the value of the prediction data on which the upper triangular masking is performed may be expressed as 0, and the remaining part may be expressed as 1.

The values of the masked prediction data may be controlled to have a probability close to zero when an arbitrary large negative value is reflected thereafter and is expressed probabilistically through a softmax function.

In the conventional transformer structure, key-query masking is performed in the encoder neural network, and upper triangular masking is performed in addition to the key-query masking in the decoder neural network. In the user churn rate prediction system of the present invention, upper triangular masking is performed in both the encoder neural network and the decoder neural network, so that the churn rate information is only provided to the user with problem information (E1, E2, ..., Ek) and the user has already submitted It can be controlled to depend on only one response information (R1, R2, ..., Rk-1).

5 is a diagram for explaining the configuration of input data according to an embodiment of the present invention.

Referring to FIG. 5 , input data may include problem information (E) and response information (R). In this case, the problem information (E) may be input to the encoder neural network of the user churn rate prediction system, and the response information (R) may be input to the decoder neural network.

The problem identification information may be a unique value assigned to each problem. The user or computer can determine what kind of problem the problem is through the problem identification information.

The problem category information may be information indicating what type of problem the corresponding problem is. For example, in the TOEIC test question, the question category may be information indicating whether the listening part or the reading part.

The start time information may be a time at which the user first encounters a given problem. That is, it may be the time the user was presented with the problem.

The input sequence position information may be information on where the corresponding problem or response is located within the entire problem or response sequence. Unlike the RNN structure, in the transformer structure, the order of input data is not indicated. In order to distinguish the order, it is necessary to separately indicate where each data is located within the entire data sequence. The input sequence position information may be embedded together with the input data, added to the embedded input data, and input to the encoding neural network and the decoding neural network.

The session location information may be information about where a problem or a response is located in the session. The input sequence location information may be counted by 1 for each problem, and when the next session is transferred, the input sequence location information may be initialized to 1 and counted again from 1.

The response accuracy information may be information indicating whether the user's response is a correct answer or an incorrect answer. For example, the index may be mapped to a vector with an index of '1' if the user's response is a correct answer, and '0' if the user's response is an incorrect answer.

The required time information may be information representing a time required for a user to solve a problem as a vector. The required time information may be expressed in seconds, minutes, hours, and the like, and for a time exceeding a predetermined time (for example, 300 seconds), it may be determined that the time (300 seconds) has been consumed.

The time-solving information may be information regarding whether the user responded within the time limit suggested by the domain expert. If true, it has an index of '1', otherwise '0', and this index can be mapped to a vector.

Bounce rate information may be information indicating whether a user has left while solving a corresponding problem, reading an explanation, or watching a lecture. If true, it has an index of '1', otherwise '0', and this index can be mapped to a vector.

The problem information (E) may include problem identification information, problem category information, start time information, input sequence position information, and session position information. However, the present invention is not limited thereto, and any one may be omitted or other information may be further added according to embodiments.

The response information R may include start time information, input sequence location information, session location information, response accuracy information, required time information, solving time information, and exit rate information. However, the present invention is not limited thereto, and any one may be omitted or other information may be further added according to embodiments.

Although not shown in the table, the problem-response log data (I) includes problem identification information, problem category information, start time information, input sequence position information, session position information, response accuracy information, required time information, time-solving information, Bounce rate information may be included. However, the present invention is not limited thereto, and any one may be omitted or other information may be further added according to embodiments.

In the user churn rate prediction system according to an embodiment of the present invention, the problem information (E) configured in the above-described data format is input to the encoder neural network of the user churn rate prediction system, and the response information (R) is input to the decoder neural network to deviate By generating rate information, there is an effect of knowing a departure rate prediction result having more improved accuracy.

6 is a table for explaining the AUC of the user churn rate prediction system according to an embodiment of the present invention.

Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment (DAS) may be a term referring to a system for predicting a user churn rate of the present invention. The area under the receiver operating characteristic curve (AUC) is a value representing the ratio of the correct prediction to the overall prediction. AUC is a value indicating sensitivity according to specificity, and the higher the AUC, the higher the prediction accuracy.

Referring to FIG. 6 , it can be confirmed that the AUC of the DAS is high compared with various artificial neural network models (LSTM, GRU).

In particular, when the sequence size is 5 (DAS-5), the AUC is 0.7379, and when the sequence size is 25 (DAS-25), the AUC is 0.6856, and when the sequence size is 5 (DAS-5), the accuracy is higher than that of the DAS-25. It can be seen that is high.

7 is a graph for explaining AUC of a user churn rate prediction system according to a sequence size of input data.

As input data, problem information and response information may be composed of various sequence sizes. 7 is a graph showing AUC according to sequence size.

Referring to FIG. 7 , when the sequence size is 5, the AUC is 0.7379, and it can be seen that a user churn rate prediction system having the highest accuracy can be implemented.

8 is a graph for explaining the AUC of the user churn rate prediction system according to the format of the input data.

The abbreviation of FIG. 8 may be understood with reference to FIG. 5 described above. Specifically, Id is the problem identification information, c is the problem category information, st is the start time information, p is the input sequence position information, sp is the session position information, r is the response accuracy information, et is the time spent information, iot may indicate in-time solution information, and d may indicate bounce rate information.

Referring to FIG. 5 , in the Base, the problem information input to the encoder neural network includes problem identification information, problem category information, and input sequence position information, and the response information input to the decoder neural network includes response accuracy information and input sequence position information. The AUC when inclusive may be 0.6346.

In this case, when start time information is further added to the problem information and start time information, intra-time solution information, and required time information are sequentially added to the response information, AUC may be 0.6653, 0.6819, and 0.7017, respectively.

Here, when session location information is added to problem information and session location information and bounce rate information are added to response information, AUC may be 0.7379.

Summarizing the above, the problem information includes problem identification information, problem category information, start time information, input sequence location information, and session location information, and the response information includes start time information, input sequence location information, session location information, It can be seen that when response accuracy information, required time information, time-solving information, and bounce rate information are included, it is possible to implement a user bounce rate prediction system having the highest accuracy.

9 is a flowchart illustrating an operation of a system for predicting a user churn rate according to an embodiment of the present invention.

Referring to FIG. 9 , in step S901 , the user churn rate prediction system may input problem information to the kth encoder neural network and response information to the kth decoder neural network, respectively.

The problem information may include problem identification information, problem category information, start time information, input sequence position information, and session position information. The response information may include start time information, input sequence location information, session location information, response accuracy information, required time information, solving time information, and exit rate information. However, the present invention is not limited thereto, and some information may be omitted or other information may be further added according to embodiments.

In step S903, the user churn rate prediction system may generate attention information by reflecting the weight in the problem information, and may generate query data by reflecting the weight in the response information.

Specifically, the problem information may reflect a weight to the problem information through self-attention. Self-attention may be an operation in which a weight is given to a part to be considered important in specific data itself, and this is reflected back to oneself.

In response information, not only self-attention, but also multi-head attention is performed based on the attention information, so that a weight may be reflected.

In operation S905 , the query data output from the first response information processing unit of the kth decoder neural network may be input to the second response information processing unit. The query data may be prediction data output from the first response information processing unit.

In step S907 , the user churn rate prediction system may train the user churn rate prediction system by using the attention information as a weight for the query data of the second response information processing unit.

delete

Since encoder neural networks and decoder neural networks can be stacked as many as N, the above process can be repeated for all the stacked encoder and decoder neural networks until a series of operations including self-attention and multi-head attention are completed.

In step S911, the user churn rate prediction system may output the churn rate information from the learned user churn rate prediction system.

This is an inference process, which processes input data according to the weight determined in the learning process, and outputs information about the churn rate indicating the probability that the user will deviate while solving the problem, while listening to or watching an explanation or lecture. can

10 is a flowchart for explaining in detail the operation of the problem information processing unit or the response information processing unit according to an embodiment of the present invention.

Referring to FIG. 10 , in step S1001, each component may receive a query, key, and value for problem information or response information. Each value may be a value expressed as a vector, or a value divided according to a role used.

In step S1003, each configuration may generate a plurality of head values for each of a query, a key, and a value.

In step S1005, each configuration generates a weight from a plurality of query head values and a plurality of key head values, and in step S1007, a key-query masking (key-query masking) and upper triangular masking (upper triangular masking). An action may be performed.

Key-query masking may optionally be an operation that prevents attention from being performed by imposing a penalty on a value without a value (zero padding). A value of prediction data on which key-query masking is performed may be expressed as 0, and the remaining part may be expressed as 1.

Upper triangular masking may be an operation of preventing attention from being performed on information corresponding to a future location (position) for prediction of a churn rate for the next problem.

For example, it may be an operation of masking that attention is performed on a problem that the user has not yet solved in order to prevent the prediction of the bounce rate information from a problem that the user has not yet solved. Like the key-query masking, the value of the prediction data on which the upper triangular masking is performed may be expressed as 0, and the remaining part may be expressed as 1.

Thereafter, in step S1009 , prediction data may be generated by applying the masked weight to the plurality of value head values.

The prediction data generated by the problem information processing unit may be attention information, the prediction data generated by the first response information processing unit may be query data, and the prediction data generated by the second response information processing unit may be departure rate information.

The user bounce rate prediction system according to the present invention may have improved performance by using an optimized input data format and using upper triangular masking for both the encoder neural network and the decoder neural network of the transformer structure.

Embodiments of the present invention published in the present specification and drawings are merely provided for specific examples in order to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. It will be apparent to those of ordinary skill in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention may be implemented in addition to the embodiments disclosed herein.

5: User Bounce Rate Prediction System
10, 30: embedding execution unit
20: Encoder Neural Network
21: Non-linearization execution unit
22: Problem information processing unit
40: decoder neural network
41: first response information processing unit
42: second response information processing unit
43: non-linearization execution unit

Claims (8)

A method of operating a user churn rate prediction system including a plurality of encoder neural networks and a plurality of decoder neural networks, the method comprising:
Learning an artificial intelligence model to predict churn rate information, which is information about a probability of a user churning out during learning, based on problem information including session location information and user response information; and
Predicting the user's churn rate information for the input problem based on the learned artificial intelligence model;
The step of learning the artificial intelligence model is,
inputting problem information to the kth encoder neural network and inputting response information to the kth decoder neural network;
generating query data by reflecting a weight in the response information, and generating attention information by reflecting the weight in the problem information; and
using the attention information as a weight for the query data to learn the user churn rate prediction system,
The session location information is
It is information indicating the location of the problem in the session, which is a learning unit divided by time based on the time when the user's departure occurred,
The step of generating the attention information includes:
Performing upper triangular masking in the plurality of encoder neural networks and the plurality of decoder neural networks to predict a bounce rate from a problem provided to a user and response information already submitted by the user; Way.
delete delete The method of claim 1, wherein whether the user leaves the
The user churn rate prediction system, including when the screen of the mobile device is turned off or another application is turned on, and the learning program in which the user performs learning in the mobile environment is switched to a background state, or when the power of the mobile device is turned off how it works.
delete delete The method of claim 1, wherein the generating of the attention information comprises:
Optionally, a method of operating a user churn rate prediction system comprising the step of performing key-query masking, which is an operation to prevent attention from being performed by imposing a penalty on a value without a value (zero padding).
Based on the problem information including the session location information and the user's response information, the artificial intelligence model is trained to predict the churn rate information, which is information about the probability that the user will leave during learning, and based on the learned artificial intelligence model , in the user churn rate prediction system for predicting the user churn rate information for the input problem,
a kth encoder neural network that receives the problem information and generates attention information by reflecting a weight on the problem information; and
A kth decoder neural network for receiving the response information, generating query data by reflecting a weight in the response information, and learning the user churn rate prediction system by using the attention information as a weight for the query data, ,
The problem information is
It includes session location information indicating the location of the problem in the session, which is a learning unit divided according to time based on the time when the user's departure occurred,
A plurality of encoder neural networks and a plurality of decoder neural networks are
A user bounce rate prediction system that performs upper triangular masking to predict the bounce rate from the problems provided to the user and the response information already submitted by the user.

Images