CN113537623A - Method and system for dynamic prediction of service demand based on attention mechanism and multimodality - Google Patents

Abstract

The present disclosure provides a method and system for dynamic prediction of service demand based on attention mechanism and multi-modality, including: acquiring text data and image data generated during service use; and performing feature extraction on the text data and image data respectively. ; Input the extracted features into a pre-trained prediction model based on soft attention and multi-modal machine learning to realize the prediction of the user's service demand at the next moment; wherein, the soft attention and multi-modal machine learning-based prediction model The prediction model is specifically: based on the feature sharing mechanism to realize the fusion of multi-modal data features; use the soft attention mechanism to process the fused features, and input the obtained results into the pre-trained GRU network to obtain the user's service interest Feature vector representation: Based on the feature vector representation of user information and service interest, the prediction of the user's service demand at the next moment is realized through the fully connected layer.

Description

Method and system for dynamic prediction of service demand based on attention mechanism and multimodality

technical field

The present disclosure belongs to the technical field of service preference prediction, and in particular relates to a method and system for dynamic prediction of service demand based on an attention mechanism and multimodality.

Background technique

The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

In recent years, with the rapid development and maturity of new computing models such as service computing, cloud computing, and mobile edge computing, a large number of available services from different fields have appeared on the network. Popularization enables large-scale users to access services anytime and anywhere, which greatly facilitates the user's life and work. However, it is challenging for users to find the services that users need from a large number of candidate services, which affects the utilization of services and the satisfaction of users. In order to solve this problem, researchers have carried out a lot of research work on service recommendation, and have achieved rich research results, which solve the problem of service discovery to a certain extent. However, the inventor found that most of the existing service recommendation methods recommend services that users are interested in and may use by mining information between similar users or similar services, without considering the actual service needs of users, resulting in the accuracy of recommendation. needs improvement.

Service demand prediction is an important basis for improving the accuracy of service recommendation. At present, scholars at home and abroad have carried out preliminary research on service demand forecasting, and have achieved certain results. Existing service demand forecasting methods mainly include forecasting methods based on collaborative filtering (CF: Collaborative Filtering) technology, based on machine learning (ML: Machine Learning) and based on deep learning (DL: Deep Learning). Specifically, Guo et al. proposed a residual space-time network for short-term travel demand forecasting, which can capture the spatial, temporal and travel demand dependencies of travel demand, and has a good forecasting effect in travel demand forecasting. . Liu et al. proposed a deep ensemble network model based on attention mechanism, which respectively modeled the inter-channel relationship, spatial relationship and location relationship of the feature map and predicted the service demand of users. For spatiotemporal demand forecasting and competitive supply problems, Zheng et al. proposed a demand-aware path planning algorithm that considered both spatiotemporal forecasting and supply and demand states, and constructed a spatiotemporal graph convolution sequential forecasting model that Time to predict user service requests. To help service providers pre-allocate service starting points to reduce customer waiting time, Chu et al. propose a multi-scale convolutional long short-term memory network model that can predict future users by considering both temporal and spatial correlations need. Aiming at the lack of considering the privacy of users in mobile, Lu et al. proposed a user collaborative filtering method combined with privacy concern strength, and considered the relevant factors about user privacy to predict service demand. Gardino et al. used a multi-view approach to learn the connections between views to solve the demand forecasting problem between retailers and wholesalers across industries. Rob et al. used deep learning methods to alleviate the complexity brought by artificial network models, and used tourism search intensity as the only input indicator to provide tourism business personnel with demand forecasts between tourists and destinations. Xu et al. used historical water demand as data information to provide effective water demand prediction for urban water supply systems. Although the existing service demand prediction research has improved the accuracy of service recommendation to a certain extent; however, most of the existing research work is based on single-modal data, and does not consider service demand forecasting under multi-modal data.

SUMMARY OF THE INVENTION

In order to solve the above problems, the present disclosure provides a method and system for dynamic prediction of service demand based on attention mechanism and multimodality. The prediction model of multimodal machine learning realizes the accurate prediction of user service needs.

According to a first aspect of the embodiments of the present disclosure, a method for dynamic prediction of service demand based on attention mechanism and multimodality is provided, including:

Obtain text data and image data generated during the use of the service;

Perform feature extraction on the text data and image data respectively; input the extracted features into a pre-trained prediction model based on soft attention and multi-modal machine learning, so as to predict the user's service demand at the next moment;

Among them, the prediction model based on soft attention and multimodal machine learning is specifically: based on the feature sharing mechanism to realize the fusion of multimodal data features; use the soft attention mechanism to process the fused features, and obtain The results are input into the pre-trained GRU network to obtain the user's service interest feature vector representation; based on the user information features and their service interest feature vector representation, the full connection layer is used to predict the user's service demand at the next moment.

Further, the fusion of multi-modal data features is realized based on the feature sharing mechanism, specifically: inputting the extracted text features and image features into the text feature network and the image feature network respectively, and fully connecting the text features and image feature networks at each layer. The output of the layer is logically added; the image feature and the output of each fully connected layer of the text feature network are logically added, and finally the output of the text feature network and the image feature network is passed through a fully connected layer to obtain the fusion result.

Further, using the soft attention mechanism to process the fused features specifically includes: calculating the weight of the fused feature information based on the soft attention mechanism, and obtaining diverse service interest expression vectors.

Further, inputting the obtained results into a pre-trained GRU network to obtain a feature vector representation of the user's service interest, specifically: the GRU network for the service used by the user at each moment and the service used at the past moment. Learn the impact of using the service at the current moment, the learning result is stored in the hidden state vector at each moment, and output a hidden state vector at each moment to represent the learned service interest information, and then obtain the user's information at each moment. Service Use Interests.

Further, an auxiliary loss function is introduced into the GRU network, and the gap between the hidden state of the GRU at each moment and the service feature fusion vector at the next moment is calculated by the auxiliary loss function.

According to a second aspect of the embodiments of the present disclosure, there is provided an attention mechanism and multimodal service demand dynamic prediction system, including:

A data acquisition unit, which is used to acquire text data and image data generated during the use of the service;

A demand prediction unit, which is used to extract features from the text data and image data respectively; input the extracted features into a pre-trained prediction model based on soft attention and multi-modal machine learning, so as to realize the service of the user at the next moment demand forecast;

Among them, the prediction model based on soft attention and multimodal machine learning is specifically: based on the feature sharing mechanism to realize the fusion of multimodal data features; use the soft attention mechanism to process the fused features, and obtain The results are input into the pre-trained GRU network to obtain the user's service interest feature vector representation; based on the user information features and their service interest feature vector representation, the full connection layer is used to predict the user's service demand at the next moment.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the memory, the processor implements the attention-based attention when the processor executes the program Force mechanism and multimodal service demand dynamic prediction method.

According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the described attention-based mechanism and multiple A modal service demand dynamic forecasting method.

Compared with the prior art, the beneficial effects of the present disclosure are:

(1) The solution described in this disclosure considers the use of text data and image data generated during service usage, and based on the proposed Soft Attention and Multimodal Machine Learning (SAMML) model, a A dynamic prediction method of service demand based on attention mechanism and multimodality, which realizes accurate prediction of user service demand.

(2) The Soft Attention and Multimodal Machine Learning (SAMML) model proposed in the scheme firstly extracts feature vectors from text data and image data respectively and performs feature sharing to realize multimodality The fusion of data features improves the expressive ability of users and services; then, the Soft Attention mechanism is applied to process the fused feature data, and the obtained results are input into the GRU network, so that the GRU network can better Finally, the SAMML model is trained based on the user characteristics and service characteristic data, and the trained SAMML model is used to accurately predict the user's service requirements.

Advantages of additional aspects of the disclosure will be set forth in part in the description that follows, and in part will become apparent from the description below, or will be learned by practice of the disclosure.

Description of drawings

The accompanying drawings that constitute a part of the present disclosure are used to provide further understanding of the present disclosure, and the exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure.

FIG. 1 is a schematic structural diagram of the SAMML model described in Embodiment 1 of the present disclosure;

FIG. 2 is a schematic diagram of the multimodal feature fusion structure described in Embodiment 1 of the present disclosure;

3 is a schematic structural diagram of the GRU neuron described in Embodiment 1 of the present disclosure;

Figure 4(a) is the model loss value when the learning rate of the SAMML model described in Embodiment 1 of the present disclosure is 1e-2;

FIG. 4(b) is the model loss value when the learning rate of the SAMML model described in Embodiment 1 of the present disclosure is 1e-3;

Figure 4(c) is the model loss value when the learning rate of the SAMML model described in Embodiment 1 of the present disclosure is 1e-4;

FIG. 4(d) is the model loss value when the learning rate of the SAMML model described in Embodiment 1 of the present disclosure is 1e-5.

Detailed ways

The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the present disclosure. Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

The embodiments of this disclosure and features of the embodiments may be combined with each other without conflict.

Example 1:

The purpose of this embodiment is to provide a dynamic prediction method for service demand based on attention mechanism and multimodality.

A dynamic prediction method of service demand based on attention mechanism and multimodality, comprising:

Obtain text data and image data generated during the use of the service;

Perform feature extraction on the text data and image data respectively; input the extracted features into a pre-trained prediction model based on soft attention and multi-modal machine learning, so as to predict the user's service demand at the next moment;

Among them, the prediction model based on soft attention and multimodal machine learning is specifically: based on the feature sharing mechanism to realize the fusion of multimodal data features; use the soft attention mechanism to process the fused features, and obtain The results are input into the pre-trained GRU network to obtain the user's service interest feature vector representation; based on the user information features and their service interest feature vector representation, the full connection layer is used to predict the user's service demand at the next moment.

The user information feature refers to the user's gender, age, occupation, economic income and other information.

Further, the fusion of multi-modal data features is realized based on the feature sharing mechanism, specifically: inputting the extracted text features and image features into the text feature network and the image feature network respectively, and fully connecting the text features and image feature networks at each layer. The output of the layer is logically added; the image feature and the output of each fully connected layer of the text feature network are logically added, and finally the output of the text feature network and the image feature network is passed through a fully connected layer to obtain the fusion result.

Further, both the text feature network and the image feature network are composed of several fully connected layers.

Further, using the soft attention mechanism to process the fused features specifically includes: calculating the weight of the fused feature information based on the soft attention mechanism, and obtaining diverse service interest expression vectors.

Further, inputting the obtained results into a pre-trained GRU network to obtain a feature vector representation of the user's service interest, specifically: the GRU network for the service used by the user at each moment and the service used at the past moment. Learn the impact of using the service at the current moment, the learning result is stored in the hidden state vector at each moment, and output a hidden state vector at each moment to represent the learned service interest information, and then obtain the user's information at each moment. Service Use Interests.

Further, an auxiliary loss function is introduced into the GRU network, and the gap between the hidden state of the GRU at each moment and the service feature fusion vector at the next moment is calculated by the auxiliary loss function.

Specifically, in order to facilitate understanding, the solutions described in the present disclosure are described in detail below with reference to the accompanying drawings:

The present disclosure considers text data and image data generated during service usage, and provides a dynamic prediction method for service requirements based on a Soft Attention and Multimodal Machine Learning (SAMML) model. The method firstly extracts feature vectors from text data and image data respectively and performs feature sharing to realize the fusion of multimodal data features and improve the expressive ability of users and services. Then, the Soft Attention mechanism is used to process the fusion. and input the obtained results into the GRU network, so that the GRU network can better learn the user's service usage interest; finally, the SAMML model is trained based on the user characteristics and service characteristic data, and the trained SAMML is used. The model realizes accurate prediction of users' service needs.

Figure 1 shows the network structure of the SAMML model. The SAMML model consists of a multimodal data feature sharing module, a service interest extraction module and a service demand prediction module.

In the SAMML model, firstly, based on the Doc2Vec model and the ResNet model, feature vectors are extracted from the text data and image data of the user service information, respectively, and the extracted feature vectors are fused through the feature sharing module; The soft attention mechanism is used to learn the weights; then based on the feature data processed by the GRU network, the user service application expression vector is learned; finally, the SAMML model is trained based on the user features and service feature vectors, and the user's service demand prediction is realized. specific:

(1) Multimodal data feature sharing module

表示第m项的训练数据，Y_m表示用户在X_m时对应的服务需求；

为用户特征，包括用户的性别、年龄、职业等；

表示服务特征信息。每一个服务特征信息包括与服务应用相关的文本数据与图像数据，其中，

表示k服务项的特征数据，

表示k服务项的文本数据特征，

表示k服务项的图像数据特征。For the service demand prediction problem, let the training data set be T=[(X ₁ ,Y ₁ ),(X ₂ ,Y ₂ ),…,(X _m ,Y _m ),…,(X _n ,Y _n )] , n is the data size of the training dataset. in,

represents the training data of the mth item, and Y _m represents the corresponding service demand of the user at X _m ;

User characteristics, including the user's gender, age, occupation, etc.;

Indicates service characteristic information. Each service feature information includes text data and image data related to the service application, wherein,

represents the characteristic data of k service items,

represents the text data features of k service items,

Image data features representing k service items.

和图像特征序列

分别输入到M_txt和M_img网络；将文本特征

与图像特征网络M_img中每层dense的输出P_img进行逻辑相加，将图像特征

与文本特征网络M_txt中每层dense的输出P_txt进行逻辑相加。如图2所示为共享模块的网络结构示意图。In order to realize the effective fusion of different modal data features, a feature sharing mechanism is adopted to realize the correlation between multi-modal data features, thereby improving the accuracy of service demand prediction. Specifically, in the SAMML model, the feature fusion is composed of a text feature network M _txt and an image feature network M _img , and M _txt and M _img are composed of a fully connected network. Taking the user's service item k as an example, the text feature sequence of the service item k is

and image feature sequence

Input to M _txt and M _img networks respectively; the text features

It is logically added with the output P _img of each layer of dense in the image feature network M _img , and the image features

It is logically added to the output P _txt of each layer of dense in the text feature network M _txt . Figure 2 is a schematic diagram of the network structure of the shared module.

图像特征网络

其中l∈[1,c]。在经过第l层的操作后，以M_txt网络传输为例，特征融合公式如(1)式和(2)式所示：Set the number of nodes in the input layer of the feature sharing module as a, and the number of layers as c. When the feature vector passes through the lth layer, the text feature network is output.

image feature network

where l∈[1,c]. After the operation of the first layer, taking _Mtxt network transmission as an example, the feature fusion formula is shown in formulas (1) and (2):

为M_txt中经过l-1层特征共享后的文本特征向量，

表示M_txt中第l层的激活函数ReLu,

表示M_txt中第l层的权重矩阵向量，

表示M_txt中第l层的偏置值。最后，将特征共享向量

和

经过一层全连接网络输出用户服务的特征共享表达向量Feature share(Fs),Fs^k的计算公式如(3)式所示：In the above formula, · represents the dot product,

is the text feature vector after l-1 layer feature sharing in M _txt ,

represents the activation function _ReLu of the lth layer in Mtxt,

represents the weight matrix vector of the lth layer in _Mtxt ,

Represents the bias value of the lth layer in _Mtxt . Finally, share the features with the vector

and

After a layer of fully connected network outputs the feature share expression vector Feature share(Fs) of the user service, the calculation formula of ^Fsk is shown in formula (3):

表示向量拼接操作符号。where σ ₁ represents the ReLu activation function in the fully connected network, W ₁ represents the weight matrix, b ₁ represents the bias value,

Represents the vector concatenation operator symbol.

(2) Feature weight acquisition module based on Soft Attention mechanism

The Soft Attention (SA) mechanism reweights and aggregates the remaining information by selectively ignoring part of the information. All information will be re-weighted in an adaptive manner before being aggregated, so that important information can be separated. And avoid the interference of these information by unimportant information, thus improving the accuracy. The present disclosure chooses to use the SA mechanism to obtain the weight of the feature information to ensure that the feature vectors are all learned during the model training process, thereby enhancing the relevance of the expression vectors; after obtaining diverse service interest expression vectors, let the model learn in the process of learning , adjust the weight value of the influence of the user's diverse service interests on the user's service demand through the user's service sequence; then, multiply the weight and the user's diverse service interest expression vector, and input it into the GRU network to dynamically model the user The changing process of diverse service interests. The SAMML model uses the feature sharing vector Fs as the input of the soft attention mechanism, and finally calculates the weighted average of different service feature vectors according to the following operations, and intuitively analyzes the ratio between different service vectors. The steps for obtaining weights based on SA are as follows:

Step 1: Initialization. Define the attention variable z to represent the index value to be queried; z∈[1,N], N represents the total amount of user service feature items; when z=k, it indicates that the feature sharing vector Fs ^k of the kth item is selected.

Step 2: After the query vector q and the feature sharing vector Fs ^k are determined, the similarity between the query vector q and the query key Fs ^k is calculated and compared, and the probability α _k of the feature sharing vector of the kth item is calculated according to formula (4), and make a normalization adjustment.

Step 3: Perform a weighted average. When the attention distribution α _k queries the vector q, the degree of correlation between the feature sharing information of the kth item in Fs and the query vector q is obtained, and the value of Soft Attention is obtained, as shown in formula (6).

Step 4: After calculating the correlation degree of different feature sharing information, process the different feature sharing information according to the result of formula (6), and take the output as the result and input it to the GRU network in turn for the next step.

Among them, the probability vector of α _k represents the attention distribution, and ^Sk is the scoring function of attention. The present disclosure adopts the dot product model as the scoring function, and its calculation formula is shown in formula (5).

S ^k =s(Fs ^k ,q)=(Fs ^k ) ^T q (5)

(3) Service interest extraction module based on GRU

上，r_t的值越小，被写入的信息量越少。经过SA机制处理后的数据作为GRU网络的输入x。In recent years, GRU (Gate Recurrent Unit) neural network, as a variant of LSTM (Long-Short TermMemory) neural network, has been widely and successfully applied in NLP and time series data processing. Compared with LSTM, GRU network has the advantages of simple structure and fast calculation speed. To this end, the present disclosure uses the GRU network to learn the service feature sharing vector to extract the user's interest in service usage. The structure of the GRU network is shown in FIG. 3 . where r _t and z _t represent the reset gate and update gate, respectively. The update gate is used to control the degree to which the service status information of the last moment is retained in the current state. The larger the value of z _t , the more the service information of the last moment is left in the current state; the function of the reset gate is to control the service status of the last moment. How much information is written to the current candidate set

On the other hand, the smaller the value of r _t , the less information is written. The data processed by the SA mechanism is used as the input x of the GRU network.

The processing process of the service feature sharing information in the GRU network is as follows:

Step 1: Output the unit value between [0,1] through z _t according to the current state x _t and the hidden state h _t-1 of the previous moment. The specific operation is shown in formula (7):

具体操作公式如(8)式(9)式所示：Step 2: According to x _t and the hidden state h _t-1 of the previous moment, output the unit value between [0, 1] through r _t , and at the same time, the function tanh creates the candidate value vector at the moment

The specific operation formula is shown in formula (8) (9):

Step 3: Using z _t as the weight vector, the candidate vector and the output vector at the previous moment are weighted to obtain the output h _t of the GRU network. The specific operation formula is shown in (10):

表示t时刻的候选激活状态，由新输入x_t前状态h_t-1和权重W^h计算更新其值；h_t表示t时刻的激活状态，表示GRU网络中第t个隐藏状态向量，根据新的z_t的前一时刻的状态h_t-1和

的值，得到新的GRU的输出值。W^u，W^r，W^h和U^u，U^r，U^h分别表示更新门和重置门的权重矩阵，b^u，b^r，b^h分别表示更新门和重置门的偏置值。For the above formula, · represents the point multiplication operation, σ is the sigmoid function, tanh represents the tanh activation function, x _t is the input of the state at time t (time t is the input of the kth service sequence after SA processing), h _t-1 is the state function of the hidden layer at the last moment, r _t is the output of the reset gate, and the result is mapped between 0 and 1 through the sigmoid function, the closer the information is to 1, the easier it is to be retained; z _t is the output of the update gate , and map the result between 0 and 1 through the sigmoid function;

Represents the candidate activation state at time t, and its value is calculated and updated by the state h _t-1 and the weight W ^h before the new input x _t ; h _t represents the activation state at time t, and represents the t-th hidden state vector in the GRU network. The state h _t-1 at the previous moment of z _t and

value to get the output value of the new GRU. W ^u , W ^r , W ^h and U ^u , U ^r , U ^h represent the weight matrices of the update gate and the reset gate, respectively, and b ^u , ^br , and b ^h represent the bias values of the update gate and the reset gate, respectively.

The GRU network can learn the services used by the user at each moment and the impact of the services used in the past on the services used at the current moment, and store the learning results in the hidden state vector at each moment, and at each moment A hidden state vector is output to represent the learned service interest information, so that the hidden state vector h _t at each moment in the GRU network can represent the user's service usage interest at each moment. In order to improve the extraction effect of GRU on service usage interest, the present disclosure introduces an auxiliary loss function L _lf (as shown in (11)) to the GRU network, which is used to calculate the hidden state of the GRU at each moment and the service feature fusion vector at the next moment. gap between.

(4) Service demand forecast based on SAMML

其中，

表示用户信息特征向量，

表示用户最终的服务兴趣表达向量，y_i表示模型的值，表示用户下一时刻的服务需求。服务需求预测模块的预测函数如公式(12)所示：After obtaining the service interest feature expression vector h _t , based on the user's service interest feature expression vector h _t and the user information feature vector, the user's service demand at the next moment is predicted. When training the service demand prediction module, define the input data as

in,

represents the user information feature vector,

It represents the final service interest expression vector of the user, and y _i represents the value of the model, which represents the service demand of the user at the next moment. The forecasting function of the service demand forecasting module is shown in formula (12):

where σ ₁ represents the ReLu activation function, W represents the weight matrix, I _i represents the input data, and b represents the bias value.

与真实的标签值y之间距离的平均值。假设训练数据的样本数量为n，则MAE的计算公式如(13)所示。In the SAMML model, according to the service sequence used by the user, the problem of predicting the user's service demand at the next moment based on multimodal machine learning belongs to the regression problem in machine learning. For regression problems in machine learning, the commonly used loss function is the squared absolute error (MAE), which refers to the predicted value of the service demand forecasting model

Average distance from the true label value y. Assuming that the number of samples of training data is n, the calculation formula of MAE is shown in (13).

The total loss function L of the SAMML model is mainly composed of two parts, the loss function L _tag of service demand prediction and the auxiliary loss function L _tf . Both L _tag and L _tf use the MAE loss function, but the input part of MAE is different. The calculation formula of the total loss function L is shown in (14):

L=L _tag +α*L _tf (14)

Among them, α represents a hyperparameter, which is used to balance the expression of user service interest and the prediction of the model. The present disclosure adopts the Adam optimization algorithm. The service demand prediction method based on SAMML model is shown in Algorithm 1.

------------------------------------------------

Algorithm 1: Service Demand Dynamic Prediction Algorithm Based on SAMML

Stage 1: Training of the SAMML model

Input: Data//Dataset Data for model training

1. Model initialization parameters;

2. FOR i TO N DO; //N is the batch number of data volume

3. Input training data items (X _i , Y _i );

与图片特征向量

的特征共享操作；4. According to formula (1), realize the output of the l-1 (1≤l≤c) layer of the text feature learning network _Mtxt

with picture feature vector

The feature sharing operation of ;

5. Obtain the output of the l-1 (1≤l≤c) layer of the text feature learning network _Mtxt according to formula (2)

6. Repeat the above steps 4 and 5, fuse the image features with the text features, and obtain the output of the l-1 (1≤l≤c) layer of the picture

7. According to formula (3), obtain the use service expression vector Fs ^k of the user;

8. According to formula (4) (5), calculate the scoring function of the soft attention mechanism for Fs ^k , and obtain the attention distribution of the service vector;

9. According to formula (6), perform a weighted average on ^Fsk to obtain the degree of association between different services;

10. According to formulas (7) and (8), output the update gate and output gate of the GRU network;

11. According to formulas (9) and (10), calculate the expression vector h _t of the service used by the user;

12. According to formulas (11), (12), (13), (14), calculate the auxiliary function value, the loss function value, the prediction function value and the total loss;

13. Update SAMML model parameters;

14.END FOR;

15. UNTIL (until the end condition of model training is met);

Stage 2: Service Demand Forecast

16. Input and run the SAMML model according to the data I _i ;

17. Output: user's service requirements;

-------------------------------------------------- --

Further, in order to prove the effectiveness of the scheme described in the present disclosure, the following specific experiments were carried out to prove:

(1) Experimental environment and experimental data

In order to verify the effectiveness of the proposed method, the present disclosure uses the Debiasing data set 1 provided by Alibaba Cloud-Tianchi to experimentally verify the proposed method. Dataset files are stored in CSV format, encoded in UTF-8. The dataset contains more than one million records, mainly including user features, product features, and labels. Here, products are mapped to services. User features include user ID, age, gender, etc.; service features include text feature txt_vec and image feature img_vec, marking the id of the product as the tag item_id. The user data information of the user CSV file includes the following table 1:

Table 1 Debiasing data information

The experimental environment is: operating system Windows 10 Professional Edition 64-bit, CPU Intel i7 5500U, RAM 4+4GB; using Python and TensorFlow 2.0 to implement the SAMML model. The present disclosure employs squared absolute error, MAE, mean squared error, MSE, root mean ^squared error, RMSE, and R2 metrics to evaluate the performance of SAMML. The calculation formula of MAE is shown in formula (13), and the calculation formulas of MSE, RMSE and R ² are shown in formulas (15)-(17):

Among them, the smaller the values of MAE, MSE and RMSE, the higher the prediction accuracy of the model, and the ^larger the value of R2, the higher the prediction accuracy of the service demand prediction model.

(2) Model parameter settings

In the SAMML model, the purpose of feature sharing is to fuse the data feature vectors of the two modalities to improve the relevance and expressiveness of users and services. The number of network layers M of this module has a certain influence on the accuracy of the model. In order to make the SAMML model have better prediction ability, this experiment is conducted by setting different network layers. In this experiment, the initial learning rates are set to 0.001 and 0.0001 respectively, and then different network layers M are set to observe the changes of the evaluation indicators (MAE, MSE, RMSE, R ² ) of the SAMML model, and then determine the feature sharing module in the The optimal value for the number of network layers. The experimental results are shown in Table 2 and Table 3. From Table 2 and Table 3 above, it can be seen that increasing the number of layers helps to improve the prediction accuracy of the SAMML model. As the number of network layers increases, the accuracy of the model presents a normal distribution trend.

Table 2 The results of the network layer M on the SAMML model when the learning rate = 0.0001

Table 3 The results of the network layer M on the SAMML model when the learning rate = 0.00001

However, when increasing the number of network layers of the feature sharing module, more parameters need to be learned, which takes longer training time and increases the risk of overfitting. According to the experimental results, when the number of network layers is 3, each index is optimal and stable under relative conditions. Therefore, the present disclosure determines that the number of network layers of the feature sharing module is 3, and the learning rate is set to 0.0001. In the SAMML model, the number of neuron nodes in each layer of the network in the feature sharing module also has a certain influence on the model prediction. The number of neuron nodes in each layer of the network is 16, 32, 64, 128 and 256. The optimal value of neuron nodes is determined through experiments. The experimental results are shown in Table 4.

Table 4 Influence of the number of neuron nodes in the feature sharing module on the SAMML model

As can be seen from Table 4 above, when the number of neuron nodes is 16 and 64, the values of the evaluation indicators MAE, MSE, RMSE, and R ² are relatively optimal. With the increase of the number of neuron nodes, the predictability of the model is slightly improved. different. At the same time, if the number of neuron nodes in each layer of the network is too low, it will easily lead to insufficient data fitting, and too many will increase the risk of model overfitting. According to the experimental results and comprehensive comparison, the present disclosure sets the number of network nodes in each layer of the feature sharing module of the SAMML model to 64.

In the SAMML model, the Adam algorithm is used to tune the parameters of the model. The learning rate of the Adam algorithm has a great influence on the stability and learning ability of the SAMML model. In order to make the model have strong predictive ability, in this experiment, the learning rate of the SAMML model is set to 1e-2, 1e-3, 1e-4, 1e-5, train the model and save the experimental results. The experimental results are shown in Figure 4(a) to Figure 4(d).

From Figure 4(a) to Figure 4(d), it can be seen that when the learning rate is 1e-4, the fitting has been shown when using 100 epochs. As the epoch on the training set increases, the loss on the test set does not decrease. When the learning rate is 1e-5, as the epoch on the training set increases, the loss on the test set is still decreasing, which is underfitting. , until the 300 epoch is relatively flat and no longer decreases. Combined with the chart analysis, it is obvious that the smaller the epoch, the better the fitting effect, and the experimental results also show that the accuracy is higher. According to the above analysis, the present disclosure sets the learning rate as 1e-4, which is 0.0001.

(3) Model performance comparison

In order to verify the performance of the prediction models proposed in the present disclosure, the present disclosure selects four multimodal machine learning-based predictive models to compare with the methods proposed in the present disclosure. Four typical prediction models are: RBMI (Recommendation Based on Multimodal Information), Multimodal IRIS (Interest-Related Item Similarity Model Based on Multimoda), SDML (Scalable deepmultimodal learning) and IMMML (Improved Multimodal Machine Learning). This experiment uses 80% of the dataset as training data for the model and 20% of the dataset as test data for the model. The performance of each model is evaluated according to the evaluation indicators, and the experimental results are shown in Table 5.

Table 5. Performance evaluation of different models on the dataset

As can be seen from Table ⁵ above, the SAMML model outperforms other comparative models on the evaluation metrics MAE, MSE, RMSE and R2. ^On the evaluation metric R2, the SAMML model outperforms the best results among other comparison models by 3.1%; on the metrics MAE, MSE, and RMSE, it leads the suboptimal results by 2.18%, 2.63%, and 2.73%, respectively. From the comparison results in the above table, it can be concluded that the SAMML model proposed in the present disclosure reduces the difference in feature vector expression between multimodalities by introducing a soft attention mechanism, and improves the prediction accuracy of user service requirements.

In order to better predict the service requirements of users, the present disclosure proposes a dynamic prediction method for service requirements based on soft attention and multimodal machine learning. This method first fuses the multi-dimensional service features of user services through the feature sharing module to enhance the relevance of user services; then introduces the Soft-Attention mechanism, so that the model can dynamically change the weight to change the impact on user service requirements; Finally, according to the multimodal feature expression vector of user information and service, the user's service demand is predicted through the fully connected network. Extensive experimental tests are carried out based on a large number of real data sets, and the superiority of the method proposed in this disclosure is verified by comparing with other typical multimodal models.

Embodiment 2:

The purpose of this embodiment is to provide a dynamic prediction system for service demand based on attention mechanism and multimodality.

A dynamic prediction system for service demand based on attention mechanism and multimodality, including:

A data acquisition unit, which is used to acquire text data and image data generated during the use of the service;

A demand prediction unit, which is used to extract features from the text data and image data respectively; input the extracted features into a pre-trained prediction model based on soft attention and multi-modal machine learning, so as to realize the service of the user at the next moment demand forecast;

Among them, the prediction model based on soft attention and multimodal machine learning is specifically: based on the feature sharing mechanism to realize the fusion of multimodal data features; use the soft attention mechanism to process the fused features, and obtain The results are input into the pre-trained GRU network to obtain the user's service interest feature vector representation; based on the user information features and their service interest feature vector representation, the full connection layer is used to predict the user's service demand at the next moment.

In further embodiments, there is also provided:

An electronic device includes a memory, a processor, and computer instructions stored on the memory and executed on the processor, and when the computer instructions are executed by the processor, the method described in the first embodiment is completed. For brevity, details are not repeated here.

It should be understood that, in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate array FPGA or other programmable logic devices , discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory may include read-only memory and random access memory and provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in the first embodiment is completed.

The method in the first embodiment can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, detailed description is omitted here.

The method and system for dynamic prediction of service demand based on attention mechanism and multi-modality provided by the above embodiments can be realized and have broad application prospects.

The above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (10)

1. A dynamic prediction method for service demand based on attention mechanism and multi-mode is characterized by comprising the following steps:

acquiring text data and image data generated in the service use process;

respectively extracting the characteristics of the text data and the image data; inputting the extracted features into a pre-trained prediction model based on soft attention and multi-modal machine learning to realize prediction of service requirements of the user at the next moment;

the prediction model based on soft attention and multi-modal machine learning specifically comprises: realizing the fusion of multi-modal data features based on a feature sharing mechanism; processing the fused features by using a soft attention mechanism, and inputting an obtained result into a pre-trained GRU network to obtain service interest feature vector representation of a user; and based on the user information characteristics and the service interest characteristic vector representation thereof, the service demand of the user at the next moment is predicted through the full connection layer.

2. The method according to claim 1, wherein the feature-sharing mechanism is used to implement fusion of multi-modal data features, specifically: respectively inputting the extracted text features and image features into a text feature network and an image feature network, and logically adding the text features and the output of each full-connection layer of the image feature network; and logically adding the image characteristic and the output of each full-connection layer of the text characteristic network, and finally passing the output of the text characteristic network and the image characteristic network through one full-connection layer to obtain a fusion result.

3. The method as claimed in claim 1, wherein the text feature network and the image feature network are composed of a plurality of fully connected layers.

4. The method according to claim 1, wherein the fused features are processed by using a soft attention mechanism, specifically: and calculating the weight of the fused feature information based on a soft attention mechanism, and obtaining diversified service interest expression vectors.

5. The method according to claim 1, wherein the obtained result is input to a pre-trained GRU network to obtain a service interest feature vector representation of the user, specifically: the GRU network learns the service used by the user at each moment and the influence of the service used at the past moment on the service used at the current moment, the learning result is stored in the hidden state vector at each moment, and a hidden state vector is output at each moment to represent the learned service interest information, so that the service use interest of the user at each moment is obtained.

6. The method as claimed in claim 1, wherein an auxiliary penalty function is introduced into the GRU network, and the difference between the hidden state of the GRU at each time and the service feature fusion vector at the next time is calculated by the auxiliary penalty function.

7. An attention-based and multi-modal dynamic prediction system for service demand, comprising:

a data acquisition unit for acquiring text data and image data generated during service use;

a demand prediction unit for performing feature extraction on the text data and the image data, respectively; inputting the extracted features into a pre-trained prediction model based on soft attention and multi-modal machine learning to realize prediction of service requirements of the user at the next moment;

the prediction model based on soft attention and multi-modal machine learning specifically comprises: realizing the fusion of multi-modal data features based on a feature sharing mechanism; processing the fused features by using a soft attention mechanism, and inputting an obtained result into a pre-trained GRU network to obtain service interest feature vector representation of a user; and based on the user information characteristics and the service interest characteristic vector representation thereof, the service demand of the user at the next moment is predicted through the full connection layer.

8. The system according to claim 7, wherein the feature-sharing mechanism is used to implement fusion of multi-modal data features, specifically: respectively inputting the extracted text features and image features into a text feature network and an image feature network, and logically adding the text features and the output of each full-connection layer of the image feature network; and logically adding the image characteristic and the output of each full-connection layer of the text characteristic network, and finally passing the output of the text characteristic network and the image characteristic network through one full-connection layer to obtain a fusion result.

9. An electronic device comprising a memory, a processor and a computer program stored and executed on the memory, wherein the processor implements a method for dynamic prediction of service demand based on attention mechanism and multi-modality as claimed in any one of claims 1 to 6 when executing the program.

10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the program when executed by a processor implements a method for dynamic prediction of service demand based on attentional mechanisms and multi-modalities according to any of claims 1-6.

Images