CN114969487A - Course recommendation method and device, computer equipment and storage medium - Google Patents

Abstract

The present application discloses a course recommendation method, device, computer equipment and storage medium. The method includes determining the current state of the target user according to the first historical course browsing data of the target user; strengthening the learning network according to the current state and the pre-trained target model to obtain candidate course categories that meet the screening conditions, in which the target reinforcement learning network model takes the course category to which the online course belongs as the action space, and the output quantity of the output action space is the same as the total number of course categories; corresponding to each candidate course category Filter the set number of target courses in the online courses and push them to the target users. Using this method, the processing form of the reinforcement learning network model is adopted to achieve the effect that the online courses recommended to users can bring long-term benefits to the network teaching platform; at the same time, the dimensionality reduction processing of the output action space in the reinforcement learning network model is realized. , which ensures the effective recommendation of online courses to the user and improves the user experience.

Description

A course recommendation method, device, computer equipment and storage medium

technical field

The present application relates to the technical field of information recommendation, and in particular, to a course recommendation method, apparatus, computer equipment and storage medium.

Background technique

With the rapid development and popularization of Internet technology, digital online learning methods have been increasingly accepted by the public, and the number of online courses for users to learn in online teaching platforms has also exploded. Faced with such a huge amount of information, it is difficult for users to quickly search for courses they are interested in or want to learn.

At present, the platform side often solves the above problems by actively or passively recommending courses to users. Most of the existing course recommendations are implemented by correlation modeling based on the historical operation data of users on the online teaching platform, so as to predict the courses that users are interested in and recommend. This modeling method only considers the users. Short-term preferences and benefits ignore the long-term benefits of the entire platform and cannot match the platform's interests. However, the existing recommendation methods considering long-term benefits have the problem that they cannot be applied to large-scale online course recommendation.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present application provide a course recommendation method, apparatus, computer equipment, and storage medium, which can implement effective recommendation for large-scale online courses on the basis of ensuring long-term platform benefits.

In a first aspect, an embodiment of the present application provides a course recommendation method, including:

Determine the current state of the target user according to the first historical course browsing data of the target user;

According to the current state and the pre-trained target reinforcement learning network model, a candidate course category that satisfies the screening conditions is obtained, wherein the target reinforcement learning network model takes the course category to which the online course belongs as the action space, and the output of the action space is The number of outputs is the same as the total number of said course categories;

A set number of target courses are selected from the online courses corresponding to each candidate course category and pushed to the target user.

Further, the step of dividing the course category to which the online course belongs includes:

Obtain the second historical course browsing data of each selected user from the message queue, and form a course browsing sequence corresponding to each of the users;

Taking each said course browsing sequence as a to-be-processed sentence, the course vector of each online course is obtained through the word vector division model to form a course vector set;

The course vector set is clustered to obtain the output number of clusters, and the cluster center vector of each cluster is correspondingly determined as a course category.

Further, determining the current state of the target user according to the historical course browsing data of the target user includes:

Perform word segmentation processing on the first historical course browsing data of the target user within the set time period, and determine the browsed course vector of the browsed course corresponding to the target user;

The current state of the target user is determined from the average vector of each of the browsed course vectors.

Further, according to the current state and the pre-trained target reinforcement learning network model, the candidate course categories that meet the screening conditions are obtained, including:

Inputting the current state into the target reinforcement learning network model, and outputting the output number of candidate vectors as an action space through the target reinforcement learning network model, wherein each of the candidate vectors respectively identifies a course category;

For each course category, through a given cumulative reward value model, combined with the current state and the current network parameters of the target reinforcement learning network model, determine the cumulative reward value of the course category;

Rank each of the course categories according to the cumulative return value, and take the course category with the first set ranking as the candidate course category.

Further, the selection of a set number of target courses from the online courses corresponding to the candidate course categories to push to the target users includes:

For each candidate course category, obtain the cluster center vector of the candidate course category;

Determine the distance value of each course vector and the cluster center vector in the cluster cluster associated with the cluster center vector;

Rank each of the course vectors according to the distance value, and take the course vector in the top second set ranking as the course to be recommended;

The target courses that satisfy the fine-grained screening conditions are selected from the courses to be recommended and pushed to the target users respectively.

Further, the training steps of the target reinforcement learning network model include:

The two reinforcement learning network models with the same network structure and different network parameters are respectively recorded as the real-time training network model and the initial reinforcement learning network model;

According to each course category identified by each of the cluster center vectors and the second historical course browsing data of each selected user, a training sample set for model training is constructed, wherein each training sample in the training sample set includes: The first state sequence of the user's current state, the target cluster center vector, the instantaneous reward value, and the second state sequence of the next state;

The loss function is fitted according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and the target reinforcement learning network model is obtained through reverse learning of the fitted loss function.

Further, the loss function fitting is performed according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and the target reinforcement learning network is obtained through reverse learning of the fitted loss function. models, including:

For each training sample, determine the current cumulative reward value of each action space vector output by the included first state sequence under the real-time training network model, and determine the maximum current cumulative reward value;

Determine the standard cumulative return value of the first state sequence relative to the target cluster center vector under the initial reinforcement learning network model;

The loss function is fitted according to the maximum current cumulative return value and the standard cumulative return value corresponding to each training sample;

The network parameters of the real-time training network model are updated according to the fitted loss function, and when the update times meet the parameter replacement period, the network parameters of the real-time training network model are used to replace the network parameters of the initial reinforcement learning network model ;

The initial reinforcement learning network model after parameter replacement is determined as the target reinforcement learning network model.

In a second aspect, an embodiment of the present application provides a course recommendation device, including:

an information determination module for determining the current state of the target user according to the first historical course browsing data of the target user;

The candidate determination module, the target obtains the candidate course category that satisfies the screening condition according to the current state and the pre-trained target reinforcement learning network model, wherein the target reinforcement learning network model uses the course category to which the online course belongs as the action space, and The output quantity of the output action space is the same as the total amount of the course category;

The target recommendation module is configured to select a set number of target courses from the online courses corresponding to the candidate course categories and push them to the target users.

In a third aspect, an embodiment of the present application further provides a computer device, including: a memory and one or more processors;

the memory for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the course recommendation method as described in the first aspect above.

In a fourth aspect, embodiments of the present application further provide a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute the course recommendation method according to the first aspect when executed by a computer processor.

A course recommendation method, device, computer equipment and storage medium provided above, the method first determines the current state of the target user according to the first historical course browsing data of the target user, and then strengthens the target pre-trained according to the current state set Learn the network model to obtain the candidate course categories that meet the screening conditions, where the output action space in reinforcement learning is identified by the course category to which the network course belongs, and the output quantity of the action space is equal to the total number of course categories; finally, the candidate courses can be obtained from the candidate courses. The category contains a certain amount of target courses selected from the online courses and pushed to the target users. The above technical solution of this embodiment mainly adopts the processing form of the reinforcement learning network model to achieve the effect that the online courses recommended to the target users can bring long-term benefits to the online education platform; Space dimensionality reduction processing, that is, by ensuring that the output number of the output action space is only the same as the number of course categories available in the online course, to solve the problem that reinforcement learning cannot adapt to large-scale data processing, thus realizing the network course to users. The effective recommendation on the terminal improves the user experience.

Description of drawings

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

1 is a schematic flowchart of a course recommendation method provided in Embodiment 1 of the present application;

2 is a schematic flowchart of a course recommendation method provided in Embodiment 2 of the present application;

FIG. 3 is a structural block diagram of a course recommendation apparatus provided in Embodiment 3 of the present application;

FIG. 4 is a schematic structural diagram of a computer device according to Embodiment 4 of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings. It should be clear that the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application, as recited in the appended claims.

In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, Nor should it be construed to indicate or imply relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific situations. Also, in the description of the present application, unless otherwise specified, "a plurality" means two or more. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship.

Example 1

FIG. 1 is a schematic flowchart of a course recommendation method provided in Embodiment 1 of the present application, and the method is suitable for recommending an online course in an online teaching platform to a user. The method may be performed by a course recommendation apparatus, which may be implemented in hardware and/or software, and is generally integrated in computer equipment.

It should be noted that, when online online teaching is used as the application scenario of this embodiment, the computer device integrating the method provided in this embodiment can be regarded as a platform server for online online teaching. Generally, there are often thousands of online courses for users to learn on the online teaching platform, showing a large-scale amount of online courses. Only the traditional reinforcement learning method can solve the problem of long-term benefits, but considering that the output of reinforcement learning is a candidate set as recommendation information, when the original data used to determine the candidate set is large, it is not impossible to guarantee that reinforcement learning can be used in Effective implementation of information recommendations.

The method for recommending courses provided by this embodiment can effectively solve the problem that the scale of online courses cannot be recommended through reinforcement learning.

As shown in Figure 1, a course recommendation method provided by the first embodiment specifically includes the following steps:

S101. Determine the current state of the target user according to the first historical course browsing data of the target user.

In this embodiment, users with learning needs need to enter the online teaching platform through registration and login operations. The network teaching platform side can record the user information of each registered user. Generally, the online teaching platform side can recommend online courses to each online user. In this embodiment, each user who enters the online teaching interface through a login operation can be regarded as a target user, and for each target user, Corresponding course recommendation can be implemented through the method provided in this embodiment.

In this embodiment, historical course browsing data can be specifically understood as data generated when a user browses the displayed interfaces within a period of time (eg, one day, one week, or even one month) after entering the online teaching platform. Specifically, it can be data generated when browsing courses, such as which courses have been browsed, how many times a course has been browsed in a period of time, etc., wherein, different courses can be distinguished by course identification numbers. For the convenience of distinction, in this embodiment, the historical course browsing data obtained relative to the target user is recorded as the first historical course browsing data.

In this embodiment, the current state can be specifically understood as the operating state that the target user has relative to the online course learning before the next action operation when the online course learning is used as the application environment, for example, the end of the A course Browsing is equivalent to a state of the user.

Specifically, in this step, the current state can be determined by analyzing the browsing data of the first historical course corresponding to the target user. It can be known that the first historical course browsing data includes the operation information of the target user relative to the online course within a certain period of time. In this step, these operation information can be regarded as a sentence, and then the target user's operation information can be obtained by analyzing it. The keyword information within a period of time, such as the operated online course ID, is then processed and aggregated to form vector information that can represent the current state.

S102 , according to the current state and the pre-trained target reinforcement learning network model, obtain a candidate course category that satisfies the screening condition.

In this embodiment, this step is equivalent to the operation of determining the candidate courses required for course recommendation through the reinforcement learning network model. What needs to be known is that reinforcement learning can be specifically understood as in an application environment, in the form of intelligent learning, the actions to be performed can be continuously selected according to the current state of the environment, so as to ensure the number of rewards obtained by performing the selected actions. maximum. In each state, the action selection is performed and the specific execution that guarantees the maximum number of rewards can be realized by the reinforcement learning network model.

The reinforcement learning network model preferably used in this embodiment is a deep reinforcement learning (Deep Q-Learning, DQN) network model, and the reinforcement learning network model formed after learning through the set training samples is recorded as the target reinforcement learning network model. Generally, the input data of the reinforcement learning network model is a current state value, and the output data is the action space data related to the application environment, and each action space represents the data information of a candidate execution action. In principle, the output quantity of the action space output by the target reinforcement learning network model is equivalent to the number of executable actions in the application environment, but this mode is more suitable for scenarios including small-scale executable actions.

In this embodiment, in principle, each online course included in the online teaching platform is equivalent to a candidate action that can be performed by a user. Because the number of online courses is relatively large, it is impossible to directly use each online course as a target reinforcement learning. Each executable candidate action output in the network model. In this embodiment, the network courses are divided into course categories, and each course category is used as the action space output by the target reinforcement learning network model. That is, the target reinforcement learning network model uses the course category to which the online course belongs as the action space, and the output quantity of the output action space is the same as the total amount of the course category.

Specifically, in this step, the current state of the target user may be used as an input data of the target reinforcement learning network model, and multiple different action space data information may be output, and the data information of each action space represents a types of courses. After that, each output course category can be regarded as a candidate set, from which the candidate course categories that meet the conditions can be screened. Among them, the filtering conditions can be set by the cumulative reward values corresponding to each action space output by the target reinforcement learning network model. Generally, for the setting of screening conditions, the greedy algorithm strategy is often used to determine the action space corresponding to the optimal cumulative reward value as the result of the coarse-grained screening in this step. This embodiment takes into account the maximum value determined based on the greedy algorithm strategy. The optimal solution cannot guarantee the diversity of the screening results. It is preferable to consider adopting a non-greedy algorithm to ensure the diversity of the coarse-grained screening results in this step. For example, consider taking multiple course categories with the highest cumulative return value as candidate course categories.

It should be noted that, in this embodiment, the online courses can be clustered and divided according to the user's course browsing habits in the form of clustering processing, and the cluster center vector of each cluster after the clustering can be regarded as a course. Category data information.

S103: Screen a set number of target courses from the online courses corresponding to the candidate course categories and push them to the target user.

In this embodiment, each candidate course category has one or more online courses to which it belongs, and these online courses can be used as candidate courses for course recommendation. Exemplarily, in this step, a certain amount of target courses can be selected from the candidate course set formed by each candidate course category through a certain screening strategy and pushed to the target user.

Specifically, for the screening of target courses, each candidate course included in the candidate course category may be analyzed, and an online course with high importance or weight value is used as the target course. In this step, the above-mentioned target course screening operation can be performed for each candidate course category, and the screened target course rankings can be pushed to the target users in no particular order, so as to display the first-level information of each target course on the target user's client superior.

A course recommendation method provided in the first embodiment of the present application mainly adopts the processing form of the reinforcement learning network model to achieve the effect that the online courses recommended to target users can bring long-term benefits to the online education platform; The dimensionality reduction processing of the output action space in the network model, that is, by ensuring that the output number of the output action space is only the same as the number of course categories available in the network course, to solve the problem that reinforcement learning cannot adapt to large-scale data processing, thus It realizes the effective recommendation of online courses to the user, and improves the user experience.

As an optional embodiment of the embodiment of the present application, on the basis of the foregoing embodiment, the step of dividing the course category to which the online course belongs may include:

It can be understood that the premise of determining candidate course categories by using the target reinforcement learning network model in this embodiment is to pre-determine which course categories exist as long as the online courses that users currently tend to browse in the online teaching platform. Thus, this optional embodiment provides a specific implementation of class classification of courses according to user's course browsing behavior data.

a1) Acquire the second historical course browsing data of each selected user from the message queue, and form a course browsing sequence corresponding to each of the users.

Among them, the message queue can be specifically understood as a cache queue that dares to cache the behavior data of each user generated when the user interacts with the backend. In this step, the historical course browsing data corresponding to each registered user or each user who has been selected to participate in the course category division within a certain period of time can be obtained from the set message queue, which is recorded as the second historical course in this embodiment. Browse data.

In this step, by analyzing and extracting the second historical course browsing data of each user, sequence information containing only the course ID of each online course browsed by the user can be formed, which is recorded as the course browsing sequence corresponding to the user, and each The course browsing sequence corresponding to the user is equivalent to a sentence that can be processed by word segmentation.

b1) Taking each of the course browsing sequences as a sentence to be processed, the course vector of each online course is obtained through a word vector partition model to form a course vector set.

Wherein, the word vector division model is preferably a correlation model word2vec for generating word vectors. In this step, each course browsing sequence can be regarded as a to-be-processed sentence as the input data of the word vector division model, and the vector set output by the word vector division model can be obtained. The course vector, the vector set including each course vector is also preferably recorded as the course vector set.

c1) Perform clustering processing on the course vector set to obtain the output number of clusters, and determine the cluster center vector of each cluster as a course category accordingly.

In this embodiment, the course vector set can be clustered by the K-means clustering algorithm, the K value of the formed cluster can be determined by the inflection point method, and the determined K value can be used as the present embodiment The number of outputs of the action space output by the target reinforcement learning network model in the example. Among them, the process of determining the K value by the inflection point method can be described as:

Search for all possible values of K within a certain range, and for each possible value, use the possible value to perform clustering processing to obtain the corresponding clustering results, and then use the error square sum calculation formula to collect the values under each K value. The clustering results are subjected to the calculation of the sum of squares of the errors. Among them, the calculation formula of the sum of squares of errors is expressed as:

其中，k为所有可能取值的最大取值，本实施例从1开始进行K值设定，具体以i来表示K取值的变化，其中，C为K值选定i时所对应聚类结果的聚类簇集合，每个p代表空间中的一个点，m_i代表K值选定i时的一个聚类中心。

Among them, k is the maximum value of all possible values. In this embodiment, the K value is set from 1. Specifically, i is used to represent the change of the value of K, where C is the corresponding cluster when the value of K is selected. The resulting cluster set, each p represents a point in the space, and m _i represents a cluster center when i is selected by the value of K.

Through the above formula, an error square sum can be obtained for each K value, and each error square sum can be connected by a line, and the K value with the largest slope change on the connection line can be used as the optimal K value.

In this embodiment, after the optimal K value is determined, the K value can be used as the output quantity of this embodiment, and the output quantity of clusters can be obtained, and the cluster center vector in each cluster can be regarded as a representation Vector information for a course category. Thus, the division of the required course categories in this embodiment is achieved.

It should be noted that another premise of determining the candidate course category by using the target reinforcement learning network model in this embodiment is: it is also necessary to ensure that the adopted target reinforcement learning network model is a pre-trained network model. This embodiment also provides Another optional embodiment to implement reinforcement learning network model training.

Specifically, as another optional embodiment of the embodiments of the present application, on the basis of the above optional embodiments, the training steps of the target reinforcement learning network model can be expressed in this embodiment as:

a2) The two reinforcement learning network models with the same network structure and different network parameters are recorded as the real-time training network model and the initial reinforcement learning network model, respectively.

Through the relevant analysis of reinforcement learning, it can be known that two neural network models with the same network structure need to be provided in the training process of reinforcement learning, but the network parameters of the two neural network models are not the same, and one of them can be recorded as requiring real-time The real-time training network model that is trained, the other is recorded as having done some kind of learning and can be used in real application scenarios, but still requires continuous updating of the initial reinforcement learning network model.

b2) Constructing a training sample set for model training according to each course category identified by each of the cluster center vectors and the second historical course browsing data of each selected user.

In this implementation, in order to ensure that the reinforcement learning network model obtained after training can match the course recommendation application scenario of this embodiment, the training sample set required for model training needs to be set based on the course recommendation application scenario of this embodiment. In this step, the above-mentioned cluster center vector identifying each course category in the course category division can be obtained, and the second historical course browsing data used in the above-mentioned course category division can also be obtained.

By analyzing the browsing data of the second historical course corresponding to each user, a training sample can be constructed, wherein each training sample in the training sample set includes: the first state sequence of the current state of the user, the target cluster center vector , the instantaneous reward value, and the second state sequence for the next state.

Specifically, in order to form a training sample, in this embodiment, from the perspective of parameters required in a reinforcement learning scenario, it is necessary to determine what information should be included in a training sample. Generally, the main parameters in reinforcement learning scenarios include the current state of the environment, the actions that the user can perform (such as which course to browse), the next state of the environment after the user performs the action, and the resulting actions after the user performs the action. instant return. By browsing the data through the second historical course, you can know which courses the user has browsed in the historical time period, thus forming the state sequence that the user has in the application environment, and then determining the value of each parameter in the training sample. Parameter information.

Exemplarily, assuming that the second historical course browsing data is analyzed to determine that the user has browsed four courses of course A, course B, course C and course D in sequence within a period of time, then this embodiment can construct a user based on the first three courses. The current state data of , and the next state data of the user based on the four courses.

Following the above description, in this embodiment, each course can determine the corresponding course vector information respectively, and thus form a first state sequence representing the current state of the user based on the course vector information of the first three courses. The average vector obtained by the average summation of the course vector information is regarded as the first state sequence. At the same time, a second state sequence representing the user's next state can also be formed based on the course vector information of the above four courses. Similarly, the course vector information of the above four courses can be summed to obtain.

After the fourth course (course D) is known, it is equivalent to knowing the action space to be executed when the user transitions from the current state to the next state. This embodiment does not directly use the vector data of course D as the action Instead, first determine the cluster to which the course D belongs, and then use the cluster center vector of the cluster to which it belongs as the action space information performed by the user when the user transitions from the current state to the next state, that is, the current state. The target cluster center vector in the training sample required by the embodiment.

Similarly, after the user transitions from the current state to the next state, he can also immediately feedback an instantaneous reward value according to the action operation performed by the user. Therefore, the instantaneous reward value that is fed back is also equivalent to a parameter information in a training sample. .

In this step, a corresponding training sample may be determined for each user in the manner described above.

c2) Fitting a loss function according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and obtaining a target reinforcement learning network model through reverse learning of the fitted loss function.

It can be known that the training process of the network model is equivalent to the actual output value obtained by inputting the input data in the training sample into the network model to be trained, and the standard output value in the training sample is compared in some way, and The process of reverse learning and adjusting the network parameters in the network model. Among them, the way of comparing the actual output value with the standard output value in a certain way is mainly realized by fitting the loss function.

Based on this, in this step, the current state (ie, the first state sequence) in the training sample can be used as the input data for implementing the training network model, and the action space with the largest cumulative reward value in the output action space can be used as the actual output value, combined with the training samples. The target cluster center vector given in as the standard output value is used to fit the loss function.

In the implementation of the loss function fitting in this embodiment, the setting is mainly based on the maximum cumulative reward value corresponding to the actual output value and the cumulative reward value corresponding to the standard output value under the initial reinforcement reinforcement learning network model. In the execution of training and learning again, the network parameters of the real-time training network model can be adjusted through the fitted loss function, and then the network parameters of the network model can also be adjusted after satisfying the conditions for adjusting the parameters of the initial reinforcement learning network model. Finally, the initial reinforcement learning network model after network parameter adjustment can be used as the currently available target reinforcement learning network model.

Further, in this embodiment, the above-mentioned step c2) can be implemented as the following steps:

It should be noted that the determination of the target reinforcement learning network model requires the participation of each of the above training samples, and each of the steps provided below in this optional embodiment needs to be performed for each training sample.

c21) For each training sample, determine the current cumulative reward value of each action space vector output by the included first state sequence under the real-time training network model, and determine the maximum current cumulative reward value.

In this step, the specific implementation mainly includes: first, input the first state sequence in the training sample as input data into the real-time training network model, and obtain each action space vector (equivalent to the output of the real-time training network model) output through the operation The cluster center vector corresponding to each course category formed by clustering).

After that, the current cumulative reward value of each action space vector can be obtained, and the maximum current cumulative reward value can be determined by comparing the current cumulative reward values. Among them, the current cumulative reward value can be obtained by calculating a known reward value determination function. Use the Bellman equation to determine the target value required for the loss function fit. Among them, the Bellman equation can be expressed as:

其中，Y表示损失函数拟合所需的目标值，Rt+1表示训练样本中的瞬时回报值，γ为一个预先设定的参数，Q(S_t+1,a,θ_t)表示在网络参数为θ_t的实时训练网络模型下所获得的一个动作空间由当前状态转换为下一状态时对应的累计回报值，

则在网络参数为θ_t'的初始强化学习网络模型下按照上述所确定目标动作空间确定的累计回报值。

Among them, Y represents the target value required for the loss function fitting, Rt+1 represents the instantaneous reward value in the training sample, γ is a preset parameter, and Q(S _t+1 , a, θ _t ) represents the network The cumulative reward value corresponding to the transformation of an action space from the current state to the next state obtained under the real-time training network model with parameter θ _t ,

Then, under the initial reinforcement learning network model with the network parameter θ _t ', the cumulative reward value is determined according to the target action space determined above.

c22) Determine the standard cumulative return value of the first state sequence relative to the target cluster center vector under the initial reinforcement learning network model.

The specific implementation of this step includes: inputting the first state sequence as input data into the initial reinforcement learning network model, and finding out the cluster center vector in the training sample from the output action space vector, and then obtaining its corresponding standard Cumulative return value.

c23) Fitting the loss function according to the maximum current cumulative return value and the standard cumulative return value corresponding to each training sample.

In this embodiment, the Bellman equation can be used to determine the target value required for the fitting of the loss function.

Among them, the Bellman equation can be expressed as:

则在网络参数为θ_t'的初始强化学习网络模型下按照上述所确定目标动作空间确定的标准累计回报值。其中，目标动作空间相当于上述最大累计回报值对应的动作空间，该目标动作空间也往往是训练样本中的目标聚类中心向量。Among them, the calculation of the Bellman equation is relative to each training sample, Y represents the target value required for the loss function fitting, R _t+1 represents the instantaneous reward value in the training sample, and γ is a preset value. parameter, Q(S _t+1 , a, θ _t ) represents the corresponding action space obtained under the real-time training network model with network parameter θ _t from the current state S _t to the next state S _t+1 The current cumulative return value, the maximum cumulative return value can be determined based on each current cumulative return value,

Then, under the initial reinforcement learning network model with the network parameter θ _t ', the standard cumulative reward value is determined according to the target action space determined above. Among them, the target action space is equivalent to the action space corresponding to the above maximum cumulative reward value, and the target action space is often the target cluster center vector in the training sample.

After that, the specific loss function value can be determined according to the given loss function, where the loss function can be expressed as:

其中，Q(S_t+1,a,θ_t表示在网络参数为θ_t的实时训练网络模型下所获得的目标动作空间由上一状态转换为当前状态S_t时对应的上一累计回报值，Y为上述确定的目标值，n为训练样本的个数，上述表达式主要拟合实时训练网络模型的实际值值和目标值Y之间的均方误差。

Among them, Q(S _t+1 , a, θ _t represents the previous cumulative reward value when the target action space obtained under the real-time training network model with network parameter θ _t is converted from the previous state to the current state S _t , Y is the target value determined above, n is the number of training samples, the above expression mainly fits the mean square error between the actual value of the real-time training network model and the target value Y.

c24) Update the network parameters of the real-time training network model according to the fitted loss function, and when the number of updates satisfies the parameter replacement period, replace the initial reinforcement learning network model with the network parameters of the real-time training network model Network parameters.

Specifically, the reverse training of the real-time training network model can be performed through the mean square error, so as to adjust the network parameters to update the network model. In this step, in addition to updating the network parameters of the real-time training network model in real time, the statistics of the number of updates are also performed in real time. When the cumulative value of the number of updates reaches the parameter replacement period, it is necessary to use the network for training the network model in real time at that moment. parameters to replace the network parameters of the initial reinforcement learning network model, so as to realize the update of the initial reinforcement learning network model.

Wherein, the preferred parameter replacement cycle in this embodiment is that the number of updates is accumulated from 0 to a quantitative value, and the quantitative value can be determined according to historical experience, such as 50 times.

c25) Determine the initial reinforcement learning network model after parameter replacement as the target reinforcement learning network model.

The above-mentioned optional embodiments of this embodiment specifically provide the realization of the course category division of the network course, and the training realization of the reinforcement learning network model with the course category formed by the division as the action space dimension. The division of course categories can effectively reduce the scale of online courses to be recommended, thereby reducing the action space dimension in reinforcement learning, and ensuring the effective application of reinforcement learning in large-scale data recommendation scenarios.

Embodiment 2

FIG. 2 is a schematic flowchart of a course recommendation method provided in Embodiment 2 of the present application. This embodiment is based on the above-mentioned embodiment. In this embodiment, the target user can be determined according to the historical course browsing data of the target user. The current state is specifically expressed as: performing word segmentation on the first historical course browsing data of the target user within the set time period, and determining the browsed course vector of the browsed courses corresponding to the target user; The average vector of browsing course vectors determines the current state of the target user.

At the same time, in this embodiment, obtaining the candidate course category that meets the screening conditions according to the current state and the pre-trained target reinforcement learning network model may be specifically expressed as: inputting the current state into the target reinforcement learning network model, Output the output number of candidate vectors as the action space through the target reinforcement learning network model, wherein each of the candidate vectors respectively identifies a course category; for each course category, through a given cumulative reward model, combined with The current state and the current network parameters of the target reinforcement learning network model determine the cumulative reward value of the course category; rank each of the course categories according to the cumulative reward value, and take the course category in the first set ranking as the Candidate course category.

In addition, in this embodiment, the acquisition of candidate course categories that meet the screening conditions according to the current state and the pre-trained target reinforcement learning network model can also be specifically expressed as: for each candidate course category, obtain the cluster center of the candidate course category vector; determine the distance value of each course vector in the cluster cluster associated with the cluster center vector and the cluster center vector; rank each of the course vectors according to the distance value, and the course vector in the first second set ranking will be As the to-be-recommended course; select a target course that satisfies the fine-grained screening condition from each of the to-be-recommended courses and push it to each of the target users respectively.

As shown in FIG. 2 , a course recommendation method provided in Embodiment 2 of the present application specifically includes the following operations:

S201. Perform word segmentation on the first historical course browsing data of the target user within a set time period, and determine a browsed course vector of the browsed course corresponding to the target user.

Exemplarily, the related model word2vec of word vectors can also be used to implement analysis processing, thereby obtaining the course vectors of all online courses that the user has browsed in the set time period, which is recorded as the browsed course vector in this step.

S202: Determine the current state of the target user from the average vector of the browsed course vectors.

This step is equivalent to an implementation of the current state, that is, an average vector representing the current state of the target user is obtained by calculating the average value of each of the browsed course vectors.

S203. Input the current state to the target reinforcement learning network model, and output the output number of candidate vectors as an action space through the target reinforcement learning network model.

This step is equivalent to the specific application of the target reinforcement learning network model, which can output multiple action spaces corresponding to the input current state, each action space is represented in the form of a vector, and each vector is recorded as a candidate vector, wherein each candidate vector A course category is respectively identified, that is, an output action space identifies a course category. Therefore, the output quantity is equivalent to the K value determined when the above-mentioned clustering algorithm is used to classify the courses.

S204. For each course category, determine the cumulative reward value of the course category by combining the current state and the current network parameters of the target reinforcement learning network model with a given cumulative reward value model.

It can be known that each course category represented by the cluster center vector can be obtained through the division of the above course categories, and this step is equivalent to the operation of each course category. The specific determination of the cumulative reward value in this step may preferably be calculated by using the cumulative reward value function described above. The required information is the candidate vector (cluster center vector) corresponding to the course category, the current network parameters of the target reinforcement learning network model, and the next state corresponding to the reinforcement learning process.

S205. Rank each of the course categories according to the cumulative return value, and take the course category at the top first set ranking as the candidate course category.

Through the above steps, the cumulative reward value corresponding to each course category can be determined. In this step, each cumulative reward value can be ranked, so that multiple course categories with the highest ranking can be selected as candidate course categories, so as to ensure that all Gain variety in coarse-grained screening results. Wherein, the first set noun may preferably be 2, that is, the course category ranked in the top two.

The following S206 to S209 give the specific implementation of fine-grained screening of target courses

S206, for each candidate course category, obtain the cluster center vector of the candidate course category.

It can be known that this step and the following S207 and S208 are operations relative to each candidate course category. One candidate course category is also equivalent to the clustering center of a cluster. This step can obtain the cluster with the center. Class center vector, that is, the candidate vector of each candidate course category relative to the target reinforcement learning network model.

S207: Determine the distance value between each course vector in the cluster associated with the cluster center vector and the cluster center vector.

In this embodiment, a cluster includes at least one online course belonging to the cluster center, and each online course is represented by a corresponding course vector. In this step, the distance value from each course vector to the cluster center vector can be calculated.

S208. Rank each of the course vectors according to the distance value, and use the course vector in the first and second set ranking as the course to be recommended.

In this step, the distance values obtained above can be ranked, so that multiple top-ranked course vectors can be obtained. In this step, the top 20 ranked courses are selected as the courses to be recommended, that is, the second set noun is preferably 20.

S209: Select target courses that satisfy the fine-grained screening conditions from the courses to be recommended and push them to the target users respectively.

In this step, the courses to be recommended corresponding to each candidate course category can be aggregated, and the recommended courses can be sorted again according to the given fine-grained filtering conditions, thereby selecting an appropriate number of online courses as target courses. Among them, the fine-grained filter conditions can be selected according to specific application scenarios, and after the selection, the sorting reference dimension for sorting the recommended courses can be determined.

In an implementation manner given in this embodiment, 4 online courses may be randomly selected from the courses to be recommended corresponding to each candidate course category as target courses and pushed to the target user. The pushed target courses can be displayed on the home page interface of the user client for the user to choose to browse.

A course recommendation method provided by the second embodiment of the present invention embodies the process of determining the current state of the target user, the process of determining the category of candidate courses, and the process of screening the target courses. The method provided in this embodiment mainly adopts the processing form of the reinforcement learning network model to achieve the effect that the online courses recommended to the target users can bring long-term benefits to the online education platform; dimensionality reduction processing, that is, by ensuring that the output number of the output action space is only the same as the number of course categories available in the online course, to solve the problem that reinforcement learning cannot adapt to large-scale data processing, thus realizing the network course to the user. Effective recommendations improve user experience.

Embodiment 3

FIG. 3 is a structural block diagram of a course recommendation apparatus provided in Embodiment 3 of the present application, and the apparatus is suitable for recommending online courses in an online teaching platform to users. The apparatus may be implemented in hardware and/or software, and is typically integrated in computer equipment. As shown in FIG. 3 , the apparatus includes: an information determination module 31 , a candidate determination module 32 and a target recommendation module 33 .

The information determination module 31 is used for determining the current state of the target user according to the first historical course browsing data of the target user;

Candidate determination module 32, the target obtains candidate course categories that meet the screening conditions according to the current state and the pre-trained target reinforcement learning network model, wherein the target reinforcement learning network model uses the course category to which the online course belongs as the action space, And the output quantity of the output action space is the same as the total amount of the course category;

The target recommendation module 33 is configured to select a set number of target courses from the online courses corresponding to each candidate course category and push them to the target user.

A course recommendation device provided by the third embodiment mainly adopts the processing form of the reinforcement learning network model to achieve the effect that the online courses recommended to target users can bring long-term benefits to the online education platform; The dimensionality reduction processing of the output action space in the model, that is, by ensuring that the output number of the output action space is only the same as the number of course categories available in the online course, to solve the problem that reinforcement learning cannot adapt to large-scale data processing, thus achieving It can effectively recommend online courses to the user, and improve the user experience.

Further, the device may also include: a course classification and division module,

The course classification and division module can be used for:

Obtain the second historical course browsing data of each selected user from the message queue, and form a course browsing sequence corresponding to each of the users;

Taking each said course browsing sequence as a to-be-processed sentence, the course vector of each online course is obtained through the word vector division model to form a course vector set;

The course vector set is clustered to obtain the output number of clusters, and the cluster center vector of each cluster is correspondingly determined as a course category.

Further, the information determination module 31 can be specifically used for:

Perform word segmentation processing on the first historical course browsing data of the target user within the set time period, and determine the browsed course vector of the browsed course corresponding to the target user;

The current state of the target user is determined from the average vector of each of the browsed course vectors.

Further, the candidate determination module 32 can be specifically used for:

Inputting the current state into the target reinforcement learning network model, and outputting the output number of candidate vectors as an action space through the target reinforcement learning network model, wherein each of the candidate vectors respectively identifies a course category;

For each course category, through a given cumulative reward value model, combined with the current state and the current network parameters of the target reinforcement learning network model, determine the cumulative reward value of the course category;

Rank each of the course categories according to the cumulative return value, and take the course category with the first set ranking as the candidate course category.

Further, the target recommendation module 33 can be specifically used for:

For each candidate course category, obtain the cluster center vector of the candidate course category;

Determine the distance value of each course vector and the cluster center vector in the cluster cluster associated with the cluster center vector;

Rank each of the course vectors according to the distance value, and take the course vector in the top second set ranking as the course to be recommended;

The target courses that satisfy the fine-grained screening conditions are selected from the courses to be recommended and pushed to the target users respectively.

Further, the apparatus may also include a model training module, wherein the model training module may include:

The information initialization unit is used to record the two reinforcement learning network models with the same network structure and different network parameters as the real-time training network model and the initial reinforcement learning network model;

The sample determination unit is configured to construct a training sample set for model training according to each course category identified by each of the cluster center vectors and the second historical course browsing data of each selected user, wherein the training sample set is Each training sample includes: the first state sequence of the user's current state, the target cluster center vector, the instantaneous reward value, and the second state sequence of the next state;

The target obtaining unit is used to perform loss function fitting according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and obtain the target reinforcement learning through reverse learning of the fitted loss function network model.

Further, the target obtaining unit can be specifically used for:

For each training sample, determine the current cumulative reward value of each action space vector output by the included first state sequence under the real-time training network model, and determine the maximum current cumulative reward value;

Determine the standard cumulative return value of the first state sequence relative to the target cluster center vector under the initial reinforcement learning network model;

The loss function is fitted according to the maximum current cumulative return value and the standard cumulative return value corresponding to each training sample;

The network parameters of the real-time training network model are updated according to the fitted loss function, and when the update times meet the parameter replacement period, the network parameters of the real-time training network model are used to replace the network parameters of the initial reinforcement learning network model ;

The initial reinforcement learning network model after parameter replacement is determined as the target reinforcement learning network model.

Embodiment 4

FIG. 4 is a schematic structural diagram of a computer device according to Embodiment 4 of the present application. The computer equipment includes: a processor 40 , a memory 41 , a display screen 42 , an input device 43 and an output device 44 . The number of processors 40 in the computer device may be one or more, and one processor 40 is taken as an example in FIG. 4 . The number of memories 41 in the computer device may be one or more, and one memory 41 is taken as an example in FIG. 4 . The processor 40 , the memory 41 , the display screen 42 , the input device 43 and the output device 44 of the computer device may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 4 . In an embodiment, the computer device may be a computer, a notebook, a smart tablet, or the like.

As a computer-readable storage medium, the memory 41 can be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the computer equipment described in any embodiment of the present invention (for example, information in the course recommendation device). A determination module 31, a candidate determination module 32, and a target recommendation module 33). The memory 41 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the device, and the like. In addition, memory 41 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory 41 may further include memory located remotely from processor 40, which may be connected to the device through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The display screen 42 may be a display screen 42 with a touch function, which may be a capacitive screen, an electromagnetic screen or an infrared screen. Generally speaking, the display screen 42 is used for displaying data according to the instruction of the processor 40, and is also used for receiving touch operations acting on the display screen 42, and sending corresponding signals to the processor 40 or other devices.

The input device 43 can be used for receiving input digital or character information, and generating key signal input related to user setting and function control of the display device, and can also be a camera for acquiring images and a pickup device for acquiring audio data. The output device 44 may include audio equipment such as speakers. It should be noted that the specific composition of the input device 43 and the output device 44 can be set according to actual conditions.

The processor 40 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 41 , that is, to implement the above-mentioned course recommendation method.

The computer device provided above can be used to execute the course recommendation method provided by any of the above embodiments, and has corresponding functions and beneficial effects.

Embodiment 5

Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute a course recommendation method when executed by a computer processor, including:

Determine the current state of the target user according to the first historical course browsing data of the target user;

According to the current state and the pre-trained target reinforcement learning network model, a candidate course category that satisfies the screening conditions is obtained, wherein the target reinforcement learning network model takes the course category to which the online course belongs as the action space, and the output of the action space is The number of outputs is the same as the total number of said course categories;

A set number of target courses are selected from the online courses corresponding to each candidate course category and pushed to the target user.

Of course, a storage medium containing computer-executable instructions provided by an embodiment of the present invention, the computer-executable instructions of which are not limited to the operations of the course recommendation method as described above, and can also execute the course recommendation method provided by any embodiment of the present invention. The related operations in the system have corresponding functions and beneficial effects.

From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general-purpose hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation manner . Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in a computer-readable storage medium, such as a floppy disk of a computer , read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including several instructions to make a computer device (which can be a robot, a personal computer, server, or network device, etc.) to execute the course recommendation method described in any embodiment of this application.

It is worth noting that the units and modules included in the above-mentioned course recommendation device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; The specific names are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application.

It should be understood that various parts of this application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Note that the above are only preferred embodiments of the present application and applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present application. The scope is determined by the scope of the appended claims.

Claims (10)

1. A course recommendation method, comprising:

determining the current state of a target user according to first historical course browsing data of the target user;

obtaining candidate course classes meeting screening conditions according to the current state and a pre-trained target reinforcement learning network model, wherein the target reinforcement learning network model takes the course class to which the network course belongs as an action space, and the output quantity of the output action space is the same as the total quantity of the course classes;

and screening a set number of target courses from the network courses corresponding to the candidate course categories and pushing the target courses to the target user.

2. The method as claimed in claim 1, wherein the step of dividing the class of the network lesson comprises:

obtaining second historical course browsing data of the selected users from the message queue, and forming course browsing sequences corresponding to the users;

taking each course browsing sequence as a sentence to be processed, and obtaining a course vector of each network course through a word vector division model to form a course vector set;

and clustering the course vector set to obtain the clustering clusters of the output quantity, and correspondingly determining the clustering center vector of each clustering cluster as a course category.

3. The method as claimed in claim 1, wherein the determining the current status of the target user according to the historical lesson browsing data of the target user comprises:

performing word segmentation on first historical course browsing data of the target user in a set time period, and determining a browsed course vector of a browsed course corresponding to the target user;

and determining the current state of the target user by the average vector of the browsed course vectors.

4. The method as claimed in claim 1, wherein said obtaining candidate course classes satisfying the filtering condition according to the current state and the pre-trained target reinforcement learning network model comprises:

inputting the current state into the target reinforcement learning network model, and outputting the output quantity of candidate vectors as an action space through the target reinforcement learning network model, wherein each candidate vector respectively identifies a course type;

determining the accumulated return value of each course category through a given accumulated return value model and by combining the current state and the current network parameters of the target reinforcement learning network model;

and ranking the course categories according to the accumulated return value, and taking the course category with the first set ranking as a candidate course category.

5. The method as claimed in claim 2, wherein the step of screening a set number of target courses from the network courses corresponding to each of the candidate course categories comprises:

aiming at each candidate course category, acquiring a clustering center vector of the candidate course category;

determining a distance value between each course vector in a cluster associated with the cluster center vector and the cluster center vector;

ranking the course vectors according to the distance value, and taking the course vector with the second previous set name as a course to be recommended;

and selecting target courses meeting fine-grained screening conditions from the courses to be recommended and pushing the target courses to the target users respectively.

6. The method of claim 2, wherein the training step of the target reinforcement learning network model comprises:

respectively recording two reinforcement learning network models with the same network structure and different network parameters as a real-time training network model and an initial reinforcement learning network model;

constructing a training sample set for model training according to the course categories identified by the clustering center vectors and the second historical course browsing data of the selected users, wherein each training sample in the training sample set comprises: the method comprises the steps of obtaining a first state sequence of a current state of a user, a target clustering center vector, an instantaneous return value and a second state sequence of a next state;

and performing loss function fitting according to output results of the training samples under the real-time training network model and the initial reinforcement learning network model respectively, and obtaining a target reinforcement learning network model through reverse learning of the fitted loss function.

7. The method according to claim 6, wherein the performing a loss function fitting according to the output results of the training samples under the real-time training network model and the initial reinforcement learning network model, respectively, and obtaining the target reinforcement learning network model through reverse learning of the fitted loss function comprises:

for each training sample, determining the current accumulated return value of each action space vector output by the first state sequence under the real-time training network model, and determining the maximum current accumulated return value;

determining a standard accumulated return value of the first state sequence relative to the target clustering center vector under the initial reinforcement learning network model;

performing loss function fitting according to the maximum current accumulated return value and the standard accumulated return value corresponding to each training sample;

updating the network parameters of the real-time training network model according to the fitted loss function, and replacing the network parameters of the initial reinforcement learning network model with the network parameters of the real-time training network model when the updating times meet a parameter replacement period;

and determining the initial reinforcement learning network model after parameter replacement as the target reinforcement learning network model.

8. A course recommending apparatus, comprising:

the information determining module is used for determining the current state of a target user according to first historical course browsing data of the target user;

the candidate determining module is used for acquiring candidate course classes meeting screening conditions by a target according to the current state and a pre-trained target reinforcement learning network model, wherein the target reinforcement learning network model takes the course class to which the network course belongs as an action space, and the output quantity of the output action space is the same as the total quantity of the course classes;

and the target recommending module is used for screening a set number of target courses from the network courses corresponding to the candidate course categories and pushing the target courses to the target user.

9. A computer device, comprising: a memory and one or more processors;

the memory for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

10. A storage medium containing computer-executable instructions for performing the method of claims 1-7 when executed by a computer processor.

Images