CN110738314B - Click rate prediction method and device based on deep migration network - Google Patents

Abstract

The invention discloses a click rate prediction method and a click rate prediction device based on a depth migration network, wherein the click rate prediction device is used for realizing the method, and the method comprises the steps of discretizing continuous fields; preprocessing discrete features, converting the discrete features into feature id codes, and obtaining a mapping dictionary; converting the characteristic id code into a characterization vector by using a Glove model as an initialization parameter of the depth migration network embedded layer; inputting the feature id code into a depth migration network for training; discretizing the test sample, and mapping the discrete features into feature id codes by using a mapping dictionary M; and inputting the feature id code into a depth migration network to predict the click rate. The invention optimizes the click rate prediction method, improves the prediction accuracy and simultaneously keeps the low predicted time delay.

Description

A method and device for predicting click rate based on deep migration network

Technical Field

The present invention relates to the field of big data processing, and in particular to a click rate prediction method and device based on a deep migration network.

Background Art

With the popularization of the Internet, everything in life is closely related to the Internet: shopping on Taobao and JD.com, ordering takeout on Meituan and Ele.me, watching movies on Tencent Video and iQiyi. The large number of clicks of users on the Internet has accumulated a large amount of valuable data for Taobao, JD.com, Meituan and other platforms. These valuable data force these platforms to invest resources to make these data generate visible value, such as using these data to calculate the estimated scenarios of advertising or recommendation systems.

The main goal of computational advertising is to integrate information from advertisers, ad space providers, and users to deliver more accurate advertising, thereby improving advertisers’ advertising effectiveness, ad space revenue, and user experience, achieving a win-win situation for all parties. In this link, the most important part is how to deliver accurate advertising, and the technology used is various methods of click-through rate estimation (a field for predicting the probability of users clicking on ads). And because the data in the computational advertising field is highly sparse and huge in magnitude, the simplest linear model such as logistic regression was widely used at first. However, since the click-through rate prediction scenario needs to consider multiple objects such as users and advertisements at the same time, the combination of features is much more important than considering each feature independently. The factor decomposition model (FM) that was developed later represents features with representation vectors and expresses the combination information between features through the inner product of the representation vectors. Great progress has been made. Recently, deep learning models have achieved great success in various fields, so that deep learning models based on neural networks have gradually been applied to the field of click-through rate prediction to make up for the shortcomings of FM models that do not include feature combinations of the second order or above. Although the effect of deep learning models is much better than that of FM models, the computational complexity is higher, and their application in massive data scenarios such as computational advertising that require real-time prediction results is greatly limited.

In general, the CTR prediction method has been continuously optimized and has achieved significant improvement in results, but the low latency requirement that needs to be considered in actual use has gradually been ignored.

Summary of the invention

The main purpose of the present invention is to propose a click rate prediction method based on a deep migration network, aiming to overcome the above problems.

To achieve the above object, the present invention proposes a click rate prediction method based on a deep migration network, comprising the following steps:

S10 discretizes the continuous fields of the training samples to obtain discrete features of the training samples;

S20 creates a unique feature id index code for each discrete feature of the training sample, and establishes a mapping dictionary M of the discrete features according to the mapping relationship between the discrete features of the training sample and their feature id index codes;

S30 counts the co-occurrence frequencies of feature id index codes in different samples to create a feature co-occurrence frequency matrix, converts the feature id index codes into a representation vector matrix of the feature co-occurrence frequency matrix through a Glove model, and uses the representation vector as an initialization parameter of an Embedding layer of a deep migration network;

S40 inputs the feature id index code into the deep migration network to obtain the cross entropy loss of the predicted click rate, and uses the back propagation algorithm to update all parameters of the deep migration network, wherein the deep migration network includes an embedding layer Embedding for converting the feature id index code into a corresponding representation vector, a factor decomposition FM network for performing an inner product on the corresponding representation vector to obtain an FM predicted click rate, a shared perceptron network for performing a nonlinear transformation on the corresponding representation vector to obtain an abstract representation vector, a lightweight perceptron network for inputting the abstract representation vector to obtain a lightweight predicted click rate, and a deep perceptron network for inputting the abstract representation vector to obtain a deep layer predicted click rate;

S50 discretizes the test sample to obtain discrete features of the test sample, and matches and maps the discrete features of the test sample based on the mapping dictionary M to obtain a feature id index code of the test sample;

S60 inputs the feature ID index code of the test sample into the deep migration network for prediction to obtain the click rate prediction of the test sample.

Preferably, the deep migration network includes an embedding layer, a factor decomposition FM network, a shared perceptron network, a lightweight perceptron network and a deep perceptron network, and S40 includes:

S401 inputs the feature ID index code into the embedding layer of the deep migration network to obtain a corresponding representation vector;

S402S401 inputs the feature ID index code into the embedding layer of the deep migration network to obtain the corresponding representation vector;

其中x为输入的特征id索引码，v为表示特征id索引码的表征向量，i,j∈n，i和j为不同的特征id索引码下标，n为特征id索引码总数，W_fm为FM网络线性回归项的权重参数，b_fm为FM网络线性回归项的偏置参数，<v_i,v_j>为表征向量v_i和v_j的内积运算；S402 inputs the corresponding representation vector into the factor decomposition FM network, and the factor decomposition FM network performs inner product on the corresponding representation vector to obtain the FM predicted click rate p _fm (x), wherein the inner product formula is as follows:

Where x is the input feature id index code, v is the representation vector representing the feature id index code, i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _Wfm is the weight parameter of the FM network linear regression term, _bfm is the bias parameter of the FM network linear regression term, < _vi , _vj > is the inner product operation of the representation vectors _vi and _vj ;

The corresponding representation vector is input into the shared perceptron network for nonlinear transformation to obtain the abstract representation vector: _hs = sigmoid( _Wsv + _bs ), where v is the representation vector input into the shared perceptron network, _Ws and bs are the weight parameters of the shared perceptron network, _bs is the bias parameter of the shared perceptron network, and _hs is the abstract representation vector output by the shared perceptron.

The abstract representation vector h _s is input into the deep perceptron network, and the predicted click rate p _deep (x) of the deep perceptron network is obtained through the following feedforward calculation formula:

为深层感知机网络的第l层权重参数，

为深层感知机网络的第l层偏置参数，

为第l层的输出向量，h_s为共享感知机网络的输出向量，且具体层数需人工设置；Where ReLU is the activation function,

is the weight parameter of the lth layer of the deep perceptron network,

is the bias parameter of the lth layer of the deep perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the specific number of layers needs to be set manually;

The abstract representation vector _hs is input into the lightweight perceptron network, and the predicted click rate _plight (x) of the lightweight perceptron network is obtained through the following feedforward calculation formula:

为轻量感知机网络的第l层权重参数，

为轻量感知机网络的第l层偏置参数，

为第l层的输出向量，h_s为共享感知机网络的输出向量，且层数取经验值,并且轻量感知机网络层数比深层感知机网络少；Where ReLU is the activation function,

is the weight parameter of the first layer of the lightweight perceptron network,

is the bias parameter of the lth layer of the lightweight perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the number of layers is an empirical value, and the number of layers of the lightweight perceptron network is less than that of the deep perceptron network;

S403 integrates the predicted click rates of the FM network, the lightweight perceptron network, and the deep perceptron network to calculate the click rate loss L(x; W, b). The calculation formula is as follows:

L(x;W,b)＝H(y,p _fm (x))+H(y,p _light (x))+H(y,p _deep (x))+λ||z _light (x) -z _deep (x)|| ² ,

Where H(y,p) is the cross entropy loss function commonly used in binary classification tasks, x is the input feature id index code, y is the binary classification label value of the training data, p is _pfm (x), which represents the click rate prediction value of the FM network, p is _plight (x), which represents the click rate prediction value of the lightweight perceptron network, and p is _pdeep (x), which represents the click rate prediction value of the deep network. λ is the weight value for determining the prediction error between the lightweight perceptron network and the deep network, and λ takes an empirical value. _zlight (x) is the model output value of the lightweight perceptron network before sigmoid transformation, _plight (x)=sigmoid( _zlight (x)), and _zdeep (x) is the model output value of the deep network before sigmoid transformation _{, pdeep} (x)=sigmoid( _zdeep (x)).

S404 uses a back propagation algorithm to update all parameters of the deep migration network according to the click rate loss L(x; W, b).

Preferably, the formula for the discretization process in S10 is as follows:

其中V为连续型特征的取值，D为离散化后的整数值，N为常数,

为向下取整符号，N需根据具体的特征取值范围确定。

Where V is the value of the continuous feature, D is the integer value after discretization, and N is a constant.

To round down the sign, N needs to be determined according to the specific feature value range.

Preferably, the S30 specifically includes:

S301 creates a feature co-occurrence frequency matrix C _n×n by using the co-occurrence frequencies of the training sample feature id index codes in different samples as matrix elements;

其中

为矩阵C_n×n的元素C_i,j的估计值，i,j∈n，i和j为不同的特征id索引码下标，n为特征id索引码总数，b_i和b_j为偏差项。计算基于矩阵元素C_i,j和其估计值

的误差J：S302 performs matrix decomposition on the feature co-occurrence frequency matrix C _n×n based on the matrix multiplication recovery error to obtain the matrix V _n×k parameters, and updates the matrix V _n×k parameters by gradient descent. The matrix decomposition formula is as follows:

in

is the estimated value of the element Ci _,j of the matrix C _n×n , i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _bi and _bj are bias terms. The calculation is based on the matrix element Ci _,j and its estimated value

The error J:

Ci _,j is the element of the matrix Cn _×n , _vi and _vj are the row vectors of the matrix Vn _×k , _bi and _bj are the bias terms, and the parameters of Vn _×k are updated by gradient descent with the goal of minimizing the error J;

S303 uses the row vector of the updated matrix V _n×k as the characterization vector of the corresponding feature.

The present invention also discloses a click rate prediction device based on a deep migration network, which is used to implement the above method and includes:

Discretization module, used to discretize the continuous fields of training samples to obtain discrete features of training samples;

A mapping module is used to create a unique feature ID index code for each discrete feature of a training sample, and to establish a mapping dictionary M of the discrete features according to the mapping relationship between the discrete features of the training sample and their feature ID index codes;

The feature representation module is used to count the co-occurrence frequencies of feature id index codes in different samples to create a feature co-occurrence frequency matrix, convert the feature id index codes into a representation vector matrix of the feature co-occurrence frequency matrix through the Glove model, and use the representation vector as the initialization parameter of the Embedding layer of the deep migration network;

The training module is used to input the feature ID index code into the deep migration network for training to obtain the click-through rate loss, and use the back-propagation algorithm to update all parameters of the deep migration network;

A testing module is used to discretize the test sample to obtain discrete features of the test sample, and match and map the discrete features of the test sample based on a mapping dictionary M to obtain a feature id index code of the test sample;

The prediction module is used to input the feature ID index code of the test sample into the deep migration network for training to obtain the click rate prediction of the test sample.

Preferably, the training module includes:

The embedding submodule is used to input the feature ID index code into the embedding layer of the deep migration network to obtain the corresponding representation vector;

其中x为输入的特征id索引码，v为表示特征id索引码的表征向量，i,j∈n，i和j为不同的特征id索引码下标，n为特征id索引码总数，W_fm为FM网络线性回归项的权重参数，b_fm为FM网络线性回归项的偏置参数，<v_i,v_j>为表征向量v_i和v_j的内积运算；The subnetwork prediction submodule is used to input the corresponding representation vector into the factor decomposition FM network. The factor decomposition FM network performs the inner product on the corresponding representation vector to obtain the FM predicted click rate p _fm (x), where the inner product formula is as follows:

Where x is the input feature id index code, v is the representation vector representing the feature id index code, i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _Wfm is the weight parameter of the FM network linear regression term, _bfm is the bias parameter of the FM network linear regression term, < _vi , _vj > is the inner product operation of the representation vectors _vi and _vj ;

The corresponding representation vector is input into the shared perceptron network for nonlinear transformation to obtain the abstract representation vector: _hs = sigmoid( _Wsv + _bs ), where v is the representation vector input into the shared perceptron network, _Ws and bs are the weight parameters of the shared perceptron network, _bs is the bias parameter of the shared perceptron network, and _hs is the abstract representation vector output by the shared perceptron.

The abstract representation vector h _s is input into the deep perceptron network, and the predicted click rate p _deep (x) of the deep perceptron network is obtained through the following feedforward calculation formula:

为深层感知机网络的第l层权重参数，

为深层感知机网络的第l层偏置参数，

为第l层的输出向量，h_s为共享感知机网络的输出向量，且具体层数需人工设置；Where ReLU is the activation function,

is the weight parameter of the lth layer of the deep perceptron network,

is the bias parameter of the lth layer of the deep perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the specific number of layers needs to be set manually;

The abstract representation vector _hs is input into the lightweight perceptron network, and the predicted click rate _plight (x) of the lightweight perceptron network is obtained through the following feedforward calculation formula:

为轻量感知机网络的第l层权重参数，

为轻量感知机网络的第l层偏置参数，

为第l层的输出向量，h_s为共享感知机网络的输出向量，且层数取经验值,并且轻量感知机网络层数比深层感知机网络少；Where ReLU is the activation function,

is the weight parameter of the first layer of the lightweight perceptron network,

is the bias parameter of the lth layer of the lightweight perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the number of layers is an empirical value, and the number of layers of the lightweight perceptron network is less than that of the deep perceptron network;

The integrated prediction submodule is used to integrate the predicted click rates of the FM network, the lightweight perceptron network, and the deep perceptron network to calculate the click rate loss L(x; W, b). The calculation formula is as follows:

L(x;W,b)＝H(y,p _fm (x))+H(y,p _light (x))+H(y,p _deep (x))+λ||z _light (x) -z _deep (x)|| ² ,

Where H(y,p) is the cross entropy loss function commonly used in binary classification tasks, x is the input feature id index code, y is the binary classification label value of the training data, p＝p _fm (x) is the click rate prediction value of the FM network, p＝p _light (x) is the click rate prediction value of the lightweight perceptron network, p＝p _deep (x) is the click rate prediction value of the deep network, λ is the weight value for determining the prediction error between the lightweight perceptron network and the deep network, λ takes an empirical value, z _light (x) is the model output value of the lightweight perceptron network before sigmoid transformation p _light (x)＝sigmoid(z _light (x)), z _deep (x) is the model output value of the deep network before sigmoid transformation p _deep (x)＝sigmoid(z _deep (x));

The parameter updating submodule is used to update all parameters of the deep migration network using the back-propagation algorithm according to the click-through rate loss L(x; W, b).

Preferably, the formula for discretization processing in the discrete module is as follows:

其中V为连续型特征的取值，D为离散化后的整数值，N为常数,

为向下取整符号，N需根据具体的特征取值范围确定。

Where V is the value of the continuous feature, D is the integer value after discretization, and N is a constant.

To round down the sign, N needs to be determined according to the specific feature value range.

Preferably, the feature characterization module comprises:

The feature co-occurrence submodule is used to create a feature co-occurrence frequency matrix C _n×n with the co-occurrence frequency of the training sample feature id index code in different samples as the matrix element;

其中

为矩阵C_n×n的元素C_i,j的估计值，i,j∈n，i和j为不同的特征id索引码下标，n为特征id索引码总数，b_i和b_j为偏差项，以误差最小化为目标，计算基于矩阵元素C_i,j和其估计值

的误差J：The decomposition submodule is used to perform matrix decomposition on the feature co-occurrence frequency matrix C _n×n based on the matrix multiplication recovery error to obtain the matrix V _n×k parameters, and then update the matrix V _n×k parameters by gradient descent. The matrix decomposition formula is as follows:

in

is the estimated value of the element Ci _,j of the matrix Cn _×n , i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _bi and _bj are deviation terms, with the goal of minimizing the error, the calculation is based on the matrix element Ci _,j and its estimated value

The error J:

Ci _,j is the element of the matrix Cn _×n , _vi and _vj are the row vectors of the matrix Vn _×k , _bi and _bj are the bias terms, and Vn _×k is updated by gradient descent;

The feature characterization submodule is used to use the row vector of the decomposed matrix V _n×k as the characterization vector of the corresponding feature.

Compared with the prior art, the present invention has the following beneficial effects: it makes up for the shortcoming that the FM model in the prior art does not include feature combinations of the second order or higher, utilizes the powerful transfer learning ability in the deep transfer network to allow the deep perceptron network to guide the learning of the lightweight perceptron network, thereby obtaining a lightweight perceptron network with better effect and performance, and finally integrates the FM network, the lightweight perceptron network and the deep perceptron network training to obtain the click-through rate loss, thereby optimizing the click-through rate prediction method and improving the prediction accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

FIG1 is a model architecture diagram of a deep migration network of the present invention;

The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

The present invention proposes a method for predicting click rate based on a deep migration network, comprising the following steps:

S10 discretizes the continuous fields of the training samples to obtain discrete features of the training samples;

S20 creates a unique feature id index code for each discrete feature of the training sample, and establishes a mapping dictionary M of the discrete features according to the mapping relationship between the discrete features of the training sample and their feature id index codes;

S30 counts the co-occurrence frequencies of feature id index codes in different samples to create a feature co-occurrence frequency matrix, converts the feature id index codes into a representation vector matrix of the feature co-occurrence frequency matrix through the Glove model, and trains the representation vector as a deep migration network to obtain click-through rate loss, and uses a back-propagation algorithm to update the initialization parameters of the embedding layer of the deep migration network;

S40 inputs the feature id index code into the deep migration network to obtain the cross entropy loss of the predicted click rate, and uses the back propagation algorithm to update all parameters of the deep migration network, wherein the deep migration network includes an embedding layer Embedding for converting the feature id index code into a corresponding representation vector, a factor decomposition FM network for performing an inner product on the corresponding representation vector to obtain the FM predicted click rate, a shared perceptron network for performing a nonlinear transformation on the corresponding representation vector to obtain an abstract representation vector, a lightweight perceptron network for inputting the abstract representation vector to obtain a lightweight predicted click rate, and a deep perceptron network for inputting the abstract representation vector to obtain a deep layer predicted click rate;

S50 discretizes the test sample to obtain discrete features of the test sample, and matches and maps the discrete features of the test sample based on the mapping dictionary M to obtain a feature id index code of the test sample;

S60 inputs the feature ID index code of the test sample into the deep migration network for prediction to obtain the click rate prediction of the test sample.

In the embodiment of the present invention, the present invention considers combining transfer learning with the click rate prediction method to achieve the purpose of maintaining the low latency advantage of the lightweight perceptron network while improving the prediction effect of the lightweight perceptron network. At the same time, the representation vector is initialized in combination with the Glove technology, so that the training process of the deep transfer network is more stable.

In addition to processing data before it is passed into the input component, the initial parameters of the Embedding layer inside the input component before the training phase are initialized by the representation vector obtained by the Glove technology.

The present invention makes up for the shortcoming that the FM model in the prior art does not include feature combinations above the second order, and utilizes the powerful transfer learning ability of the deep transfer network to allow the deep perceptron network to guide the learning of the lightweight perceptron network, thereby obtaining a lightweight perceptron network with better effect and performance, and finally integrating the FM network, the lightweight perceptron network and the deep perceptron network training to obtain the click-through rate loss, thereby optimizing the click-through rate prediction method and improving the prediction accuracy.

Preferably, the deep migration network includes an embedding layer Embedding that converts a feature id index code into a corresponding representation vector, a factorization FM network for performing an inner product on the corresponding representation vector to obtain an FM predicted click-through rate, a shared perceptron network for performing a nonlinear transformation on the corresponding representation vector to obtain an abstract representation vector, a lightweight perceptron network for inputting an abstract representation vector to obtain a lightweight predicted click-through rate, and a deep perceptron network for inputting an abstract representation vector to obtain a deep predicted click-through rate.

Preferably, the S40 includes:

S401 inputs the feature ID index code into the embedding layer of the deep migration network to obtain a corresponding representation vector;

其中x为输入的特征id索引码，v为表示特征id索引码的表征向量，i,j∈n，i和j为不同的特征id索引码下标，n为特征id索引码总数，W_fm为FM网络线性回归项的权重参数，b_fm为FM网络线性回归项的偏置参数，<v_i,v_j>为表征向量v_i和v_j的内积运算；S402 inputs the corresponding representation vector into the factor decomposition FM network, and the factor decomposition FM network performs inner product on the corresponding representation vector to obtain the FM predicted click rate p _fm (x), wherein the inner product formula is as follows:

Where x is the input feature id index code, v is the representation vector representing the feature id index code, i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _Wfm is the weight parameter of the FM network linear regression term, _bfm is the bias parameter of the FM network linear regression term, < _vi , _vj > is the inner product operation of the representation vectors _vi and _vj ;

The corresponding representation vector is input into the shared perceptron network for nonlinear transformation to obtain the abstract representation vector: _hs = sigmoid( _Wsv + _bs ), where v is the representation vector input into the shared perceptron network, _Ws and bs are the weight parameters of the shared perceptron network, _bs is the bias parameter of the shared perceptron network, and _hs is the abstract representation vector output by the shared perceptron.

The abstract representation vector h _s is input into the deep perceptron network, and the predicted click rate p _deep (x) of the deep perceptron network is obtained through the following feedforward calculation formula:

为深层感知机网络的第l层权重参数，

为深层感知机网络的第l层偏置参数，

为第l层的输出向量，h_s为共享感知机网络的输出向量，且具体层数需人工设置；Where ReLU is the activation function,

is the weight parameter of the lth layer of the deep perceptron network,

is the bias parameter of the lth layer of the deep perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the specific number of layers needs to be set manually;

The abstract representation vector _hs is input into the lightweight perceptron network, and the predicted click rate _plight (x) of the lightweight perceptron network is obtained through the following feedforward calculation formula:

为轻量感知机网络的第l层权重参数，

为轻量感知机网络的第l层偏置参数，

为第l层的输出向量，h_s为共享感知机网络的输出向量，且层数取经验值,并且轻量感知机网络层数比深层感知机网络少；Where ReLU is the activation function,

is the weight parameter of the first layer of the lightweight perceptron network,

is the bias parameter of the lth layer of the lightweight perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the number of layers is an empirical value, and the number of layers of the lightweight perceptron network is less than that of the deep perceptron network;

S403 integrates the predicted click rates of the FM network, the lightweight perceptron network, and the deep perceptron network to calculate the click rate loss L(x; W, b). The calculation formula is as follows:

L(x;W,b)＝H(y,p _fm (x))+H(y,p _light (x))+H(y,p _deep (x))+λ||z _light (x) -z _deep (x)|| ² ,

Where H(y,p) is the cross entropy loss function commonly used in binary classification tasks, x is the input feature id index code, y is the binary classification label value of the training data, p is _pfm (x), which represents the click rate prediction value of the FM network, p is _plight (x), which represents the click rate prediction value of the lightweight perceptron network, and p is _pdeep (x), which represents the click rate prediction value of the deep network. λ is the weight value for determining the prediction error between the lightweight perceptron network and the deep network, and λ takes an empirical value. _zlight (x) is the model output value of the lightweight perceptron network before sigmoid transformation, _plight (x)=sigmoid( _zlight (x)), and _zdeep (x) is the model output value of the deep network before sigmoid transformation _{, pdeep} (x)=sigmoid( _zdeep (x)).

S404 uses a back propagation algorithm to update all parameters of the deep migration network according to the click rate loss L(x; W, b).

Preferably, the formula for the discretization process in S10 is as follows:

其中V为连续型特征的取值，D为离散化后的整数值，N为常数,

为向下取整符号，N需根据具体的特征取值范围确定。

Where V is the value of the continuous feature, D is the integer value after discretization, and N is a constant.

To round down the sign, N needs to be determined according to the specific feature value range.

Preferably, the S30 specifically includes:

S301 creates a feature co-occurrence frequency matrix C _n×n by using the co-occurrence frequencies of the training sample feature id index codes in different samples as matrix elements;

其中

为矩阵C_n×n的元素C_i,j的估计值，i,j∈n，i和j为不同的特征id索引码下标，n为特征id索引码总数，b_i和b_j为偏差项。计算基于矩阵元素C_i,j和其估计值

的误差J：S302 performs matrix decomposition on the feature co-occurrence frequency matrix C _n×n based on the matrix multiplication recovery error to obtain the matrix V _n×k parameters, and updates the matrix V _n×k parameters by gradient descent. The matrix decomposition formula is as follows:

in

is the estimated value of the element Ci _,j of the matrix C _n×n , i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _bi and _bj are bias terms. The calculation is based on the matrix element Ci _,j and its estimated value

The error J:

Ci _,j is the element of the matrix Cn _×n , _vi and _vj are the row vectors of the matrix Vn _×k , _bi and _bj are the bias terms, and the parameters of Vn _×k are updated by gradient descent with the goal of minimizing the error J;

S303 uses the row vector of the updated matrix V _n×k as the characterization vector of the corresponding feature.

Preferably, the training samples are data marked with click category labels, the click categories include no-click and click, the no-click category label is 0, and the click category label is 1; the test samples are data without click category labels.

The present invention also discloses a click rate prediction device based on a deep migration network, which is used to implement the above method. The method refers to the above embodiment, and the present device adopts all the technical solutions of all the above embodiments, so it at least has all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here. It includes:

Discretization module, used to discretize the continuous fields of training samples to obtain discrete features of training samples;

A mapping module is used to create a unique feature ID index code for each discrete feature of a training sample, and to establish a mapping dictionary M of the discrete features according to the mapping relationship between the discrete features of the training sample and their feature ID index codes;

The feature representation module is used to count the co-occurrence frequencies of feature id index codes in different samples to create a feature co-occurrence frequency matrix, convert the feature id index codes into a representation vector matrix of the feature co-occurrence frequency matrix through the Glove model, and use the representation vector as the initialization parameter of the Embedding layer of the deep migration network;

The training module is used to input the feature ID index code into the deep migration network for training to obtain the click-through rate loss, and use the back-propagation algorithm to update all parameters of the deep migration network;

A testing module is used to discretize the test sample to obtain discrete features of the test sample, and match and map the discrete features of the test sample based on a mapping dictionary M to obtain a feature id index code of the test sample;

The prediction module is used to input the feature ID index code of the test sample into the deep migration network for training to obtain the click rate prediction of the test sample.

Preferably, the training module includes:

The embedding submodule is used to input the feature ID index code into the embedding layer of the deep migration network to obtain the corresponding representation vector;

其中x为输入的特征id索引码，v为表示特征id索引码的表征向量，i,j∈n，i和j为不同的特征id索引码下标，n为特征id索引码总数，W_fm为FM网络线性回归项的权重参数，b_fm为FM网络线性回归项的偏置参数，<v_i,v_j>为表征向量v_i和v_j的内积运算；The subnetwork prediction submodule is used to input the corresponding representation vector into the factor decomposition FM network. The factor decomposition FM network performs the inner product on the corresponding representation vector to obtain the FM predicted click rate p _fm (x), where the inner product formula is as follows:

Where x is the input feature id index code, v is the representation vector representing the feature id index code, i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _Wfm is the weight parameter of the FM network linear regression term, _bfm is the bias parameter of the FM network linear regression term, < _vi , _vj > is the inner product operation of the representation vectors _vi and _vj ;

The corresponding representation vector is input into the shared perceptron network for nonlinear transformation to obtain the abstract representation vector: _hs = sigmoid( _Wsv + _bs ), where v is the representation vector input into the shared perceptron network, _Ws and bs are the weight parameters of the shared perceptron network, _bs is the bias parameter of the shared perceptron network, and _hs is the abstract representation vector output by the shared perceptron.

The abstract representation vector h _s is input into the deep perceptron network, and the predicted click rate p _deep (x) of the deep perceptron network is obtained through the following feedforward calculation formula:

为深层感知机网络的第l层权重参数，

为深层感知机网络的第l层偏置参数，

为第l层的输出向量，h_s为共享感知机网络的输出向量，且具体层数需人工设置；Where ReLU is the activation function,

is the weight parameter of the lth layer of the deep perceptron network,

is the bias parameter of the lth layer of the deep perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the specific number of layers needs to be set manually;

The abstract representation vector _hs is input into the lightweight perceptron network, and the predicted click rate _plight (x) of the lightweight perceptron network is obtained through the following feedforward calculation formula:

为轻量感知机网络的第l层权重参数，

为轻量感知机网络的第l层偏置参数，

为第l层的输出向量，h_s为共享感知机网络的输出向量，且层数取经验值,并且轻量感知机网络层数比深层感知机网络少；Where ReLU is the activation function,

is the weight parameter of the first layer of the lightweight perceptron network,

is the bias parameter of the lth layer of the lightweight perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the number of layers is an empirical value, and the number of layers of the lightweight perceptron network is less than that of the deep perceptron network;

The integrated prediction submodule is used to integrate the predicted click rates of the FM network, the lightweight perceptron network, and the deep perceptron network to calculate the click rate loss L(x; W, b). The calculation formula is as follows:

L(x;W,b)＝H(y,p _fm (x))+H(y,p _light (x))+H(y,p _deep (x))+λ||z _light (x) -z _deep (x)|| ² ,

Where H(y,p) is the cross entropy loss function commonly used in binary classification tasks, x is the input feature id index code, y is the binary classification label value of the training data, p＝p _fm (x) is the click rate prediction value of the FM network, p＝p _light (x) is the click rate prediction value of the lightweight perceptron network, p＝p _deep (x) is the click rate prediction value of the deep network, λ is the weight value for determining the prediction error between the lightweight perceptron network and the deep network, λ takes an empirical value, z _light (x) is the model output value of the lightweight perceptron network before sigmoid transformation p _light (x)＝sigmoid(z _light (x)), z _deep (x) is the model output value of the deep network before sigmoid transformation p _deep (x)＝sigmoid(z _deep (x));

The parameter updating submodule is used to update all parameters of the deep migration network using the back-propagation algorithm according to the click-through rate loss L(x; W, b).

Preferably, the formula for discretization processing in the discrete module is as follows:

其中V为连续型特征的取值，D为离散化后的整数值，N为常数,

为向下取整符号，N需根据具体的特征取值范围确定。

Where V is the value of the continuous feature, D is the integer value after discretization, and N is a constant.

To round down the sign, N needs to be determined according to the specific feature value range.

Preferably, the feature characterization module comprises:

The feature co-occurrence submodule is used to create a feature co-occurrence frequency matrix C _n×n with the co-occurrence frequency of the training sample feature id index code in different samples as the matrix element;

其中

为矩阵C_n×n的元素C_i,j的估计值，i,j∈n，i和j为不同的特征id索引码下标，n为特征id索引码总数，b_i和b_j为偏差项，以误差最小化为目标，计算基于矩阵元素C_i,j和其估计值

的误差J：The decomposition submodule is used to perform matrix decomposition on the feature co-occurrence frequency matrix C _n×n based on the matrix multiplication recovery error to obtain the matrix V _n×k parameters, and then update the matrix V _n×k parameters by gradient descent. The matrix decomposition formula is as follows:

in

is the estimated value of the element Ci _,j of the matrix Cn _×n , i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _bi and _bj are deviation terms, with the goal of minimizing the error, the calculation is based on the matrix element Ci _,j and its estimated value

The error J:

Ci _,j is the element of the matrix Cn _×n , _vi and _vj are the row vectors of the matrix Vn _×k , _bi and _bj are the bias terms, and Vn _×k is updated by gradient descent;

The feature characterization submodule is used to use the row vector of the decomposed matrix V _n×k as the characterization vector of the corresponding feature.

For additional understanding, the model architecture of the deep migration network is shown in Figure 1. The model has five components:

1. Input components

The input component requires data to be input in the form of discrete feature id codes, and then the feature id codes are passed to the Embedding layer to generate representation vectors, where the feature id codes must be processed into feature id codes to facilitate the Embedding layer to quickly retrieve the corresponding representation vectors. Therefore, before passing the data to the input component, continuous floating-point values such as amounts must be discretized. Specifically, long-tailed floating-point values with a wide range of values can be discretized according to the following formula:

为向下取整符号。此外对于本身就是离散型的特征也要进行处理，例如对于性别＝[男，女]，要转化成性别＝[0，1]；对于年龄＝[12，13，...，80]，要转化成年龄＝[0，1，...，68]。Where V is the value of the continuous feature, D is the integer value after discretization, and N is a constant.

In addition, the discrete features should also be processed, for example, gender = [male, female] should be converted to gender = [0, 1]; age = [12, 13, ..., 80] should be converted to age = [0, 1, ..., 68].

In addition to processing the data before it is passed into the input component, the initial parameters of the embedding layer inside the input component before the training phase are initialized by the representation vector obtained by the Glove technology. The steps for using Glove to convert the feature ID encoding into the representation vector are as follows:

S3.1: According to the common features of different samples in the data, the number of times they appear in the same sample is counted, and finally a feature co-occurrence frequency matrix C _n×n is obtained;

S3.2: Perform the following matrix decomposition on the feature co-occurrence frequency matrix:

Where V _n×k is the decomposed matrix and bias is the bias term. The error is recovered based on the following matrix multiplication:

Where Ci _,j is the element of the matrix Cn _×n , _vi and _vj are the row vectors of the matrix Vn _×k , _bi and _bj are the bias terms, and the parameters of the matrix _Vn×k are obtained by gradient descent update

S3.3: The row vectors of the matrix V _n×k after decomposition of the feature co-occurrence frequency matrix are used as the representation vectors of the corresponding features.

2.FM Network

The FM network uses the representation vector as input and the inner product of the representation vector as a feature combination method. This explicit second-order feature combination method is more efficient and has better generalization ability in scenarios such as click-through rate estimation where data is highly sparse. Compared with the perceptron network that does not include explicit feature combination, integrating the FM network into the deep transfer network can guide the embedding layer in the deep transfer network to learn better representation vectors.

3. Shared Perceptron Network

The shared perceptron network takes the representation vector as input, and uses the complex nonlinear mapping capability of the perceptron network to convert the original representation vector into a more abstract representation vector, and allows the subsequent lightweight perceptron network and deep perceptron network to share the abstract representation vector as the network input. In this most direct way of information migration, the information of the deep perceptron network can be conveniently used by the lightweight network to achieve the purpose of improving performance.

4. Deep Perceptron Network

The deep perceptron network takes the abstract representation vector transformed by the shared perceptron network as input, and further passes through more layers of nonlinear combination, so it has the ability to represent higher-order feature combinations, so it has better performance than the lightweight perceptron network. In order to transfer the information learned by the deep perceptron network to the lightweight perceptron network, we use the output click rate of the deep perceptron network to enrich the 0-1 label of whether the original data is clicked. The training sample uses data marked with click category labels, and the click category includes no click and click. The category label of no click is 0, and the category label of click is 1; the test sample is data without click category label. Compared with the original label that only provides category attribution information 1 or 0, the output click rate of the deep perceptron network can provide more information, not only to know whether the probability of one category is greater than the probability of another category, but also to know the exact probability value strength information.

5. Lightweight Perceptron Network

The lightweight perceptron network also uses the abstract representation vector converted by the shared perceptron network as input. On the one hand, it utilizes the information of the abstract representation vector through several layers of shallow nonlinear combinations to achieve the goal of more accurate prediction of click-through rate. On the other hand, it learns the information mined from the data by the deep perceptron network by fitting the predicted click-through rate of the deep perceptron network, so as to achieve the goal of improving the prediction effect while maintaining low latency for the lightweight perceptron network.

The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly/indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims (8)

1. A click rate prediction method based on a deep migration network, characterized by comprising the following steps: S10 discretizes the continuous fields of the training samples to obtain discrete features of the training samples; S20 creates a unique feature id index code for each discrete feature of the training sample, and establishes a mapping dictionary M of the discrete features according to the mapping relationship between the discrete features of the training sample and their feature id index codes; S30 counts the co-occurrence frequencies of feature id index codes in different samples to create a feature co-occurrence frequency matrix, converts the feature id index codes into a representation vector matrix of the feature co-occurrence frequency matrix through a Glove model, and uses the representation vector as an initialization parameter of an Embedding layer of a deep migration network; S40 inputs the feature ID index code into the deep migration network to obtain the cross entropy loss of the predicted click rate, and uses the back propagation algorithm to update all parameters of the deep migration network; including: S401 inputs the feature ID index code into the embedding layer of the deep migration network to obtain a corresponding representation vector; S402 inputs the corresponding representation vector into the factor decomposition FM network, and the factor decomposition FM network performs inner product on the corresponding representation vector to obtain the FM predicted click rate p _fm (x), wherein the inner product formula is as follows:

Where x is the input feature id index code, v is the representation vector representing the feature id index code, i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _Wfm is the weight parameter of the FM network linear regression term, _bfm is the bias parameter of the FM network linear regression term, ＜ _vi , _vj ＞ is the inner product operation of the representation vectors _vi and _vj ; The corresponding representation vector is input into the shared perceptron network for nonlinear transformation to obtain the abstract representation vector: _hs = sigmoid( _Wsv + _bs ), where v is the representation vector input into the shared perceptron network, _Ws and bs are the weight parameters of the shared perceptron network, _bs is the bias parameter of the shared perceptron network, and _hs is the abstract representation vector output by the shared perceptron. The abstract representation vector h _s is input into the deep perceptron network, and the predicted click rate p _deep (x) of the deep perceptron network is obtained through the following feedforward calculation formula:

Where ReLU is the activation function,

is the weight parameter of the lth layer of the deep perceptron network,

is the bias parameter of the lth layer of the deep perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the specific number of layers needs to be set manually; The abstract representation vector _hs is input into the lightweight perceptron network, and the predicted click rate _plight (x) of the lightweight perceptron network is obtained through the following feedforward calculation formula:

Where ReLU is the activation function,

is the weight parameter of the first layer of the lightweight perceptron network,

is the bias parameter of the lth layer of the lightweight perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the number of layers is an empirical value, and the number of layers of the lightweight perceptron network is less than that of the deep perceptron network; S403 integrates the predicted click rates of the FM network, the lightweight perceptron network, and the deep perceptron network to calculate the click rate loss L(x; W, b). The calculation formula is as follows: L(x;W,b)＝H(y,p _fm (x))+H(y,p _light (x))+H(y,p _deep (x))+λ||z _light (x) -z _deep (x)|| ² , Where H(y,p) is the cross entropy loss function commonly used in binary classification tasks, x is the input feature id index code, y is the binary classification label value of the training data, p is _pfm (x), which represents the click rate prediction value of the FM network, p is _plight (x), which represents the click rate prediction value of the lightweight perceptron network, and p is _pdeep (x), which represents the click rate prediction value of the deep network. λ is the weight value for determining the prediction error between the lightweight perceptron network and the deep network, and λ takes an empirical value. _zlight (x) is the model output value of the lightweight perceptron network before sigmoid transformation, _plight (x)=sigmoid( _zlight (x)), and _zdeep (x) is the model output value of the deep network before sigmoid transformation _{, pdeep} (x)=sigmoid( _zdeep (x)). S404 uses a back propagation algorithm to update all parameters of the deep migration network according to the click rate loss L(x; W, b); S50 discretizes the test sample to obtain discrete features of the test sample, and matches and maps the discrete features of the test sample based on the mapping dictionary M to obtain a feature id index code of the test sample; S60 inputs the feature ID index code of the test sample into the deep migration network for prediction, and obtains the click rate prediction of the test sample. 2. The click-through rate prediction method based on the deep migration network as described in claim 1 is characterized in that the deep migration network includes an embedding layer Embedding that converts the feature id index code into a corresponding representation vector, a factor decomposition FM network for performing an inner product on the corresponding representation vector to obtain an FM predicted click-through rate, a shared perceptron network for performing a nonlinear transformation on the corresponding representation vector to obtain an abstract representation vector, a lightweight perceptron network for inputting an abstract representation vector to obtain a lightweight predicted click-through rate, and a deep perceptron network for inputting an abstract representation vector to obtain a deep predicted click-through rate. 3. The click rate prediction method based on deep migration network according to claim 1, characterized in that the formula for discretization processing in S10 is specifically as follows:

Where V is the value of the continuous feature, D is the integer value after discretization, and N is a constant.

To round down the sign, N needs to be determined according to the specific feature value range. 4. The method for predicting click rate based on a deep migration network according to claim 1, wherein S30 specifically comprises: S301 creates a feature co-occurrence frequency matrix C _n×n using the co-occurrence frequencies of the feature ID index codes of the training samples in different samples as matrix elements; S302 performs matrix decomposition on the feature co-occurrence frequency matrix C _n×n based on the matrix multiplication recovery error to obtain the matrix V _n×k parameters, and updates the matrix V _n×k parameters by gradient descent. The matrix decomposition formula is as follows:

in

is the estimated value of the element Ci _,j of the matrix Cn _×n , i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _bi and _bj are bias terms, and the calculation is based on the matrix element Ci _,j and its estimated value

The error J:

Ci _,j is the element of the matrix Cn _×n , _vi and _vj are the row vectors of the matrix Vn _×k , _bi and _bj are the bias terms, and the parameters of Vn _×k are updated by gradient descent with the goal of minimizing the error J; S303 uses the row vector of the updated matrix V _n×k as the characterization vector of the corresponding feature. 5. The click rate prediction method based on deep migration network as described in claim 1 is characterized in that the training sample is data marked with click category labels, the click categories include no click and click, the category label of no click is 0, and the category label of click is 1; the test sample is data without click category labels. 6. A click rate prediction device based on a deep migration network, characterized by comprising: Discretization module, used to discretize the continuous fields of training samples to obtain discrete features of training samples; A mapping module is used to create a unique feature ID index code for each discrete feature of a training sample, and to establish a mapping dictionary M of the discrete features according to the mapping relationship between the discrete features of the training sample and their feature ID index codes; The feature representation module is used to count the co-occurrence frequencies of feature id index codes in different samples to create a feature co-occurrence frequency matrix, convert the feature id index codes into a representation vector matrix of the feature co-occurrence frequency matrix through the Glove model, and use the representation vector as the initialization parameter of the Embedding layer of the deep migration network; The training module is used to input the feature ID index code into the deep migration network for training to obtain the click-through rate loss, and use the back-propagation algorithm to update all parameters of the deep migration network; including: The embedding submodule is used to input the feature ID index code into the embedding layer of the deep migration network to obtain the corresponding representation vector; The subnetwork prediction submodule is used to input the corresponding representation vector into the factor decomposition FM network. The factor decomposition FM network performs the inner product on the corresponding representation vector to obtain the FM predicted click rate p _fm (x), where the inner product formula is as follows:

Where x is the input feature id index code, v is the representation vector representing the feature id index code, i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _Wfm is the weight parameter of the FM network linear regression term, _bfm is the bias parameter of the FM network linear regression term, ＜ _vi , _vj ＞ is the inner product operation of the representation vectors _vi and _vj ; The corresponding representation vector is input into the shared perceptron network for nonlinear transformation to obtain the abstract representation vector: _hs = sigmoid( _Wsv + _bs ), where v is the representation vector input into the shared perceptron network, _Ws and bs are the weight parameters of the shared perceptron network, _bs is the bias parameter of the shared perceptron network, and _hs is the abstract representation vector output by the shared perceptron. The abstract representation vector h _s is input into the deep perceptron network, and the predicted click rate p _deep (x) of the deep perceptron network is obtained through the following feedforward calculation formula:

Where ReLU is the activation function,

is the weight parameter of the lth layer of the deep perceptron network,

is the bias parameter of the lth layer of the deep perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the specific number of layers needs to be set manually; The abstract representation vector _hs is input into the lightweight perceptron network, and the predicted click rate _plight (x) of the lightweight perceptron network is obtained through the following feedforward calculation formula:

Where ReLU is the activation function,

is the weight parameter of the first layer of the lightweight perceptron network,

is the bias parameter of the lth layer of the lightweight perceptron network,

is the output vector of the lth layer, h _s is the output vector of the shared perceptron network, and the number of layers is an empirical value, and the number of layers of the lightweight perceptron network is less than that of the deep perceptron network; The integrated prediction submodule is used to integrate the predicted click rates of the FM network, the lightweight perceptron network, and the deep perceptron network to calculate the click rate loss L(x; W, b). The calculation formula is as follows: L(x;W,b)＝H(y,p _fm (x))+H(y,p _light (x))+H(y,p _deep (x))+λ||z _light (x) -z _deep (x)|| ² , Where H(y,p) is the cross entropy loss function commonly used in binary classification tasks, x is the input feature id index code, y is the binary classification label value of the training data, p＝p _fm (x) is the click rate prediction value of the FM network, p＝p _light (x) is the click rate prediction value of the lightweight perceptron network, p＝p _deep (x) is the click rate prediction value of the deep network, λ is the weight value for determining the prediction error between the lightweight perceptron network and the deep network, λ takes an empirical value, z _light (x) is the model output value of the lightweight perceptron network before sigmoid transformation p _light (x)＝sigmoid(z _light (x)), z _deep (x) is the model output value of the deep network before sigmoid transformation p _deep (x)＝sigmoid(z _deep (x)); The parameter updating submodule is used to update all parameters of the deep migration network using the back propagation algorithm according to the click-through rate loss L(x; W, b); A testing module is used to discretize the test sample to obtain discrete features of the test sample, and match and map the discrete features of the test sample based on a mapping dictionary M to obtain a feature id index code of the test sample; The prediction module is used to input the feature ID index code of the test sample into the deep migration network for training to obtain the click rate prediction of the test sample. 7. The click rate prediction device based on deep migration network according to claim 6, characterized in that the formula for discretization processing in the discretization module is as follows:

Where V is the value of the continuous feature, D is the integer value after discretization, and N is a constant.

To round down the sign, N needs to be determined according to the specific feature value range. 8. The click rate prediction device based on deep migration network according to claim 6, characterized in that the feature representation module comprises: The feature co-occurrence submodule is used to create a feature co-occurrence frequency matrix C _n×n with the co-occurrence frequency of the training sample feature id index code in different samples as the matrix element; The decomposition submodule is used to perform matrix decomposition on the feature co-occurrence frequency matrix C _n×n based on the matrix multiplication recovery error to obtain the matrix V _n×k parameters, and then update the matrix V _n×k parameters by gradient descent. The matrix decomposition formula is as follows:

in

is the estimated value of the element Ci _,j of the matrix Cn _×n , i,j∈n, i and j are different feature id index code subscripts, n is the total number of feature id index codes, _bi and _bj are deviation terms, with the goal of minimizing the error, the calculation is based on the matrix element Ci _,j and its estimated value

The error J:

Ci _,j is the element of the matrix Cn _×n , _vi and _vj are the row vectors of the matrix Vn _×k , _bi and _bj are the bias terms, and Vn _×k is updated by gradient descent; The feature characterization submodule is used to use the row vector of the updated matrix V _n×k as the characterization vector of the corresponding feature.

Images