CN116842202A - Multimedia information recommendation method and device based on multi-mode data of wearable equipment - Google Patents
Multimedia information recommendation method and device based on multi-mode data of wearable equipment Download PDFInfo
- Publication number
- CN116842202A CN116842202A CN202210306451.2A CN202210306451A CN116842202A CN 116842202 A CN116842202 A CN 116842202A CN 202210306451 A CN202210306451 A CN 202210306451A CN 116842202 A CN116842202 A CN 116842202A
- Authority
- CN
- China
- Prior art keywords
- multimedia information
- preference
- vector
- individual
- characterization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 239000013598 vector Substances 0.000 claims abstract description 135
- 238000012512 characterization method Methods 0.000 claims abstract description 76
- 238000012937 correction Methods 0.000 claims abstract description 73
- 230000007613 environmental effect Effects 0.000 claims abstract description 18
- 238000012545 processing Methods 0.000 claims description 33
- 238000013507 mapping Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000002354 daily effect Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 101000822695 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C1 Proteins 0.000 description 1
- 101000655262 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C2 Proteins 0.000 description 1
- 101000655256 Paraclostridium bifermentans Small, acid-soluble spore protein alpha Proteins 0.000 description 1
- 101000655264 Paraclostridium bifermentans Small, acid-soluble spore protein beta Proteins 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/43—Querying
- G06F16/435—Filtering based on additional data, e.g. user or group profiles
- G06F16/436—Filtering based on additional data, e.g. user or group profiles using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physiology (AREA)
- Multimedia (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a multimedia information recommendation method and device based on multi-mode data of wearable equipment, wherein the method comprises the following steps: acquiring individual multimedia information playing records, and acquiring environmental data and individual state data acquired based on wearable equipment; generating corresponding historical multimedia information characterization according to the multimedia information play record; generating a corresponding multimedia information preference vector according to the historical multimedia information representation, and generating a corresponding multimedia information preference correction vector according to the state data and the environment data; generating individual multimedia information interest vectors according to the multimedia information preference vectors and the multimedia information preference correction vectors; generating individual multimedia information interest characterization according to the multimedia information interest vector and the historical multimedia information characterization; based on the multimedia information interest characterization, multimedia information is recommended to the individual from the plurality of candidate multimedia information. The method and the device can improve the accuracy of multimedia information recommendation based on the multi-mode data.
Description
Technical Field
The application relates to the technical field of artificial intelligence and information recommendation, in particular to a multimedia information recommendation method and device based on multi-mode data of wearable equipment.
Background
In the related art, when recommending multimedia information to a user individual, a two-dimensional matrix of the preference of the multimedia information of a target user is generally required to be acquired, and based on the similarity of the preference among different users, the preference multimedia information of other users with the maximum similarity to the preference of the target user is taken as the multimedia information to be recommended; or based on the similarity between different multimedia information, taking a plurality of multimedia information with the maximum similarity with the preferred multimedia information of the target user as the multimedia information to be recommended. However, when the user preference changes with time, the speed and accuracy of the above-mentioned recommendation method may also be affected.
Disclosure of Invention
The application provides a multimedia information recommendation method and device based on multi-mode data of wearable equipment. The multi-mode data can be acquired based on the wearable equipment, so that the tracking of the change trend of the interest of the individual is realized, and the accuracy of multimedia information recommendation of the individual is improved.
In a first aspect, the present application provides a multimedia information recommendation method based on multi-modal data of a wearable device, including: acquiring individual multimedia information playing records, and acquiring environmental data acquired based on the wearable equipment and state data of the individual; generating a corresponding historical multimedia information representation according to the multimedia information playing record; generating a corresponding multimedia information preference vector according to the historical multimedia information representation, and generating a corresponding multimedia information preference correction vector according to the state data and the environment data; generating a multimedia information interest vector of the individual according to the multimedia information preference vector and the multimedia information preference correction vector; generating a multimedia information interest representation of the individual according to the multimedia information interest vector and the historical multimedia information representation; based on the multimedia information interest characterization, multimedia information is recommended from a plurality of candidate multimedia information to the individual.
In one implementation, the generating the corresponding multimedia information preference vector according to the historical multimedia information characterization includes: inputting the historical multimedia information characterization into a preset multimedia information preference weight learning model to generate the multimedia information preference vector; wherein the preference weight learning model has learned in advance to obtain a mapping relationship between the historical multimedia information characterization and the multimedia information preference vector.
In an alternative implementation, the generating the corresponding multimedia information preference correction vector according to the state data and the environment data includes: inputting the state data and the environment data into a preset environment preference correction model to generate the multimedia information preference correction vector; wherein the environment preference correction factor learning model has learned in advance to obtain the state data and a mapping relationship between the environment data and the multimedia information preference correction vector.
Optionally, cross stitch units are adopted to share information between the multimedia information preference weight learning model and the environment preference correction model.
In one implementation, the generating the individual multimedia information interest vector according to the multimedia information preference vector and the multimedia information preference correction vector includes: and multiplying the multimedia information preference vector with the multimedia information preference correction vector point to generate the individual multimedia information interest vector.
In one implementation, the recommending multimedia information from a plurality of candidate multimedia information to the individual based on the multimedia information interest characterization includes: acquiring multimedia information characterization of each candidate multimedia information; respectively carrying out similar matching on the multimedia information interest characterization and the multimedia information characterization of each candidate multimedia information to obtain the recommendation probability of each candidate multimedia information; selecting candidate multimedia information with the recommendation probability meeting a preset recommendation condition from the plurality of candidate multimedia information; and recommending the candidate multimedia information with the recommendation probability meeting the preset recommendation condition to the individual.
According to the technical scheme, the individual multimedia interest characterization considering the individual multimedia information interest drift can be obtained based on the environment data and the multi-mode data of the individual state data, so that the change of the individual multimedia information preference is tracked, and the accuracy of multimedia information recommendation is improved.
In a second aspect, the present application provides a multimedia information recommendation apparatus based on multi-modal data of a wearable device, including: the acquisition module is used for acquiring individual multimedia information playing records and acquiring environmental data acquired based on the wearable equipment and state data of the individual; the first processing module is used for generating a corresponding historical multimedia information representation according to the multimedia information playing record; the second processing module is used for generating a corresponding multimedia information preference vector according to the historical multimedia information representation and generating a corresponding multimedia information preference correction vector according to the state data and the environment data; the third processing module is used for generating a multimedia information interest vector fourth processing module of the individual according to the multimedia information preference vector and the multimedia information preference correction vector, and is used for generating a multimedia information interest representation of the individual according to the multimedia information interest vector and the historical multimedia information representation; and the recommendation module is used for recommending the multimedia information to the individual from the plurality of candidate multimedia information based on the multimedia information interest characterization.
In one implementation, the second processing module is specifically configured to: inputting the historical multimedia information characterization into a preset multimedia information preference weight learning model to generate the multimedia information preference vector; wherein the preference weight learning model has learned in advance to obtain a mapping relationship between the historical multimedia information characterization and the multimedia information preference vector.
In an alternative implementation, the second processing module is specifically configured to: inputting the state data and the environment data into a preset environment preference correction model to generate the multimedia information preference correction vector; wherein the environment preference correction factor learning model has learned in advance to obtain the state data and a mapping relationship between the environment data and the multimedia information preference correction vector.
Optionally, cross stitch units are adopted to share information between the multimedia information preference weight learning model and the environment preference correction model.
In one implementation, the third processing module is specifically configured to: and multiplying the multimedia information preference vector with the multimedia information preference correction vector point to generate the individual multimedia information interest vector.
In one implementation, the recommendation module is specifically configured to: acquiring multimedia information characterization of each candidate multimedia information; respectively carrying out similar matching on the multimedia information interest characterization and the multimedia information characterization of each candidate multimedia information to obtain the recommendation probability of each candidate multimedia information; selecting candidate multimedia information with the recommendation probability meeting a preset recommendation condition from the plurality of candidate multimedia information; and recommending the candidate multimedia information with the recommendation probability meeting the preset recommendation condition to the individual.
According to the technical scheme, the recommendation probability of the candidate multimedia information can be obtained based on the generated interest characterization of the multimedia information, the candidate multimedia information is recommended, and the accuracy of the candidate recommendation of the media information is improved.
In a third aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of multimedia information recommendation based on wearable device multimodal data as described in the first aspect.
In a fourth aspect, the present application provides a computer readable storage medium storing instructions that, when executed, cause the method according to the first aspect to be implemented.
In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of multimedia information recommendation based on wearable device multimodal data as described in the first aspect.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the application or to delineate the scope of the application. Other features of the present application will become apparent from the description that follows.
Drawings
The drawings are included to provide a better understanding of the present application and are not to be construed as limiting the application. Wherein:
fig. 1 is a flowchart of a multimedia information recommendation method based on multi-mode data of a wearable device according to an embodiment of the present application;
fig. 2 is a flowchart of another multimedia information recommendation method based on multi-mode data of a wearable device according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a multimedia information recommendation system based on multi-mode data of a wearable device according to an embodiment of the present application;
Fig. 4 is a schematic diagram of a multimedia information recommendation device based on multi-mode data of a wearable device according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Wherein, in the description of the present application, "/" means or is meant unless otherwise indicated, for example, a/B may represent a or B; "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. The first, second, etc. numbers referred to in the present application are merely for convenience of description and are not intended to limit the scope of the embodiments of the present application, nor represent the sequence.
Referring to fig. 1, fig. 1 is a flowchart of a multimedia information recommendation method based on multi-mode data of a wearable device according to an embodiment of the present application. It should be noted that the multimedia information recommendation method in the embodiment of the present application may be applied to an electronic device. As shown in fig. 1, the multimedia information recommendation method based on the multi-modal data of the wearable device may include the following steps.
Step S101, acquiring individual multimedia information playing records, and acquiring environment data and individual state data acquired based on wearable equipment.
In some embodiments of the present application, the source of the playing record of the individual multimedia information may be a wearable device (also called wearable device) worn by the individual, or may be a mobile terminal of the individual, or an external audio/video playing device such as an intelligent sound box of the individual. The wearable device may be a watch, or VR (Virtual Reality) glasses, etc. For example, taking a wearable device as an example, the wearable device may upload the multimedia information playing record played by the individual to the electronic device for storage, so that the electronic device may obtain the multimedia information playing record of the individual.
It will be appreciated that since wearable devices typically have sensors for gathering environmental data, as well as sensors for gathering biometric data. In one implementation, the wearable device may send the collected environmental data and the status data of the individual to the electronic device for storage, so that the electronic device may obtain the environmental data and the status data of the individual in which the individual is located.
It should be noted that, in some embodiments of the present application, the multimedia information may include, but is not limited to, audio and Video information, such as Music, short Video, MV (Music Video), or may also be audio news, audio novels, etc.; the environmental data may include, but is not limited to: one or more of weather, temperature, humidity, illumination, etc. of the environment in which the individual is located for a period of time; the status data of the individual may include, but is not limited to, one or more of an amount of an individual's movement in an environment in which the individual is located over a period of time, an individual's heart rate, PAI (Personal Activity Intelligence, personal functional motor index) values, individual's sleep condition, and the like.
In some embodiments of the present application, the number of entries of specific multimedia information included in the individual multimedia information play record is the same as the number of entries of each specific category of environment data and individual status data.
Taking the example that the environmental data contains temperature and the state data comprises the heart rate of an individual and the sleeping condition of the individual as an example, assuming that the acquired individual multimedia information playing record is 30 pieces, the temperature data in the acquired environmental data can be the weather condition of the past 30 days in the geographical position of the individual every day; the acquired state data of the individual may be a daily heart rate variability status of the individual and a daily sleep status over the past 30 days.
For example, based on the category of the multimedia information to be recommended, the multimedia information record of the individual playing uploaded by the wearable device in the same category can be obtained, and the environmental data and the status data of the individual collected and uploaded by the wearable device can be obtained.
Step S102, corresponding historical multimedia information characterization is generated according to the multimedia information playing record.
For example, the multimedia information corresponding to each play record can be obtained according to the play records of the multimedia information, the corresponding multimedia information characterization can be generated based on the specific characteristics of each piece of multimedia information, and the historical multimedia information characterization can be obtained by combining each piece of multimedia information characterization.
Wherein, in embodiments of the present application, the multimedia information characterization is a symbol set consisting of a plurality of symbols, wherein the plurality of symbols may characterize a plurality of different dimensional features of the multimedia information. For example, where the multimedia information is audio information, dimensional characteristics of the multimedia information may include, but are not limited to, one or more of dance level, energy, loudness, original musical level, musical instrument level, pleasure level, musical tempo, live broadcasting level, voice level, tone, pitch, etc. of the audio information.
As an example, assuming a total of M multimedia information play records, each multimedia information may generate a multimedia information representation comprising N symbols, each symbol representing a feature of a dimension of the multimedia information, the historical multimedia information representation generated from the multimedia information play records may be represented as follows.
Wherein R is multimedia information representation, R M,1 ,R M,2 ,...R M,N N symbols representing the mth piece of multimedia information.
Step S103, corresponding multimedia information preference vectors are generated according to the historical multimedia information characterization, and corresponding multimedia information preference correction vectors are generated according to the state data and the environment data.
It should be noted that, in the embodiment of the present application, the vector dimension of the generated multimedia information preference vector and the vector dimension of the multimedia information preference correction vector are the same as the number of multimedia information play records, so as to perform subsequent calculation.
As an example, assuming that there are M pieces of multimedia information play records in total, the vector dimension of the generated multimedia information preference vector and the vector dimension of the multimedia information preference correction vector are both M.
In one implementation, generating a corresponding multimedia information preference vector from a historical multimedia information representation includes: inputting the historical multimedia information characterization into a preset multimedia information preference weight learning model to generate a multimedia information preference vector; the preference weight learning model is used for learning in advance to obtain the mapping relation between the historical multimedia information characterization and the individual multimedia information preference vector.
As an example, in an embodiment of the present application, the preference weight learning model may be a Residual Network (Resnet) 50 model.
In an alternative implementation, generating a corresponding multimedia information preference correction vector from the state data and the environment data includes: inputting the state data and the environment data into a preset environment preference correction model to generate a multimedia information preference correction vector; the environment preference correction factor learning model is used for learning in advance to obtain state data and a mapping relation between the environment data and the multimedia information preference correction vector.
As an example, in an embodiment of the present application, the environmental preference correction model may be a BiLSTM (Bi-directional Long-Short Term Memory) model.
For example, the historical multimedia information representation is input into a preset multimedia information preference weight learning model, so that a corresponding multimedia information preference vector can be obtained; and inputting the state data and the environment data into a preset environment preference correction model to obtain a corresponding multimedia information preference correction vector.
Optionally, cross stitch units are adopted to share information between the multimedia information preference weight learning model and the environment preference correction model.
For example, when training the initial models of the multimedia information preference weight learning model and the environment preference correction model using the matching probability and the cross entropy between the known user history multimedia information preferences as the loss function, information exchange can be performed between the two models to learn each other to adjust the respective model coefficients. When the corresponding multimedia information preference vector is obtained through the multimedia information preference weight learning model and the corresponding multimedia information preference correction vector is obtained through the environment preference correction model, information exchange can be carried out between the two models, so that weight coefficients of the models can be adjusted in real time according to conditions, and the accuracy of output results of the models is improved.
Step S104, generating individual multimedia information interest vectors according to the multimedia information preference vector and the multimedia information preference correction vector.
In one implementation, generating an individual multimedia information interest vector from a multimedia information preference vector and a multimedia information preference correction vector, includes: and multiplying the multimedia information preference vector with the multimedia information preference correction vector point to generate an individual multimedia information interest vector.
For example, assume that the multimedia information preference vector is p= |p 1 ,P 2 ,...P M |, the multimedia information preference correction vector is c= |c 1 ,C 2 ,...C M The multimedia information interest vector may be represented as follows.
I＝P×C＝|P 1 ×C 1 ,P 2 ×C 2 ,...P M ×C M |＝|I 1 ,I 2 ,...I M |
Wherein I is a multimedia information vector, I 1 ,I 2 ,...I M Is an element in the multimedia information vector.
Step S105, generating individual multimedia information interest characterization according to the multimedia information interest vector and the historical multimedia information characterization.
For example, the multimedia information interest vector is multiplied by the historical multimedia information representation to obtain the multimedia information interest representation of the corresponding individual, which can be expressed as follows.
Wherein IR is multimedia information interest representation, IR 1 ,IR 2 ,...IR M A plurality of symbols in the multimedia information interest representation.
Step S106, recommending the multimedia information to the individual from the candidate multimedia information based on the multimedia information interest characterization.
For example, the multimedia information interest preference of the individual may be determined based on the multimedia information interest characterization, so as to obtain the similarity between each candidate multimedia information and the multimedia information interest preference of the individual, and the multimedia information is recommended to the individual from high to low according to the similarity.
By implementing the embodiment of the application, the individual multimedia interest characterization considering the individual multimedia information interest drift can be obtained based on the environment data and the multi-mode data of the individual state data, so that the change of the individual multimedia information preference is tracked, and the accuracy of the multimedia information recommendation is improved.
Referring to fig. 2, fig. 2 is a flowchart of another multimedia information recommendation method based on multi-mode data of a wearable device according to an embodiment of the present application. The embodiment of the application can obtain the recommendation probability of the candidate multimedia information based on the generated multimedia information interest characterization, thereby recommending the candidate multimedia information. As shown in fig. 2, the multimedia information recommendation method based on the multi-modal data of the wearable device may include the following steps.
Step S201, acquiring individual multimedia information playing records, and acquiring environment data and individual state data acquired based on the wearable equipment.
In the embodiment of the present application, step S201 may be implemented in any manner of each embodiment of the present application, which is not limited to this embodiment, and is not repeated.
Step S202, corresponding historical multimedia information characterization is generated according to the multimedia information playing record.
In the embodiment of the present application, step S202 may be implemented in any one of the embodiments of the present application, which is not limited to this embodiment, and is not repeated.
Step S203, corresponding multimedia information preference vectors are generated according to the historical multimedia information characterization, and corresponding multimedia information preference correction vectors are generated according to the state data and the environment data.
In the embodiment of the present application, step S203 may be implemented in any one of the embodiments of the present application, which is not limited to this embodiment, and is not described in detail.
Step S204, generating individual multimedia information interest vectors according to the multimedia information preference vectors and the multimedia information preference correction vectors.
In the embodiment of the present application, step S204 may be implemented in any one of the embodiments of the present application, which is not limited to this embodiment, and is not repeated.
Step S205, generating individual multimedia information interest characterization according to the multimedia information interest vector and the historical multimedia information characterization.
In the embodiment of the present application, step S205 may be implemented in any one of the embodiments of the present application, which is not limited to this embodiment, and is not repeated.
Step S206, the multimedia information characterization of each candidate multimedia information is obtained.
In the embodiment of the present application, step S206 may be implemented in any one of the embodiments of the present application, which is not limited to this embodiment, and is not repeated.
Step S207, the multimedia information interest characterization is respectively subjected to similar matching with the multimedia information characterization of each candidate multimedia information, and the recommendation probability of each candidate multimedia information is obtained.
For example, the multimedia information interest characterization and the multimedia information characterization of each candidate multimedia information can be subjected to cosine calculation to obtain the matching degree of each candidate multimedia information, and the matching degree of all candidate music is subjected to normalization processing to obtain the recommendation probability of each candidate multimedia information.
As an example, assuming that there are b candidate multimedia information in total, the matching degree of the a (a.ltoreq.b) th candidate multimedia information may be expressed as follows.
d a ＝(IR×r a T )/(|IR|×|r a |)
Wherein d a The matching degree of the a candidate multimedia information.
The recommendation probability of the a-th candidate multimedia information is:
wherein D is a The recommendation probability of the a candidate multimedia information is obtained.
Step S208, selecting candidate multimedia information with recommendation probability meeting preset recommendation conditions from the plurality of candidate multimedia information.
For example, the candidate multimedia information with the highest recommendation probability or a plurality of candidate multimedia information with the top recommendation probability can be selected from the plurality of candidate multimedia information to be recommended.
Step S209, recommending the candidate multimedia information with recommendation probability meeting the preset recommendation condition to the individual.
By implementing the embodiment of the application, the recommendation probability of the candidate multimedia information can be obtained based on the generated interest characterization of the multimedia information, and the accuracy of the candidate recommendation of the media information is improved.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a multimedia information recommendation system based on multi-mode data of a wearable device according to an embodiment of the present application. As shown in fig. 3, the multimedia information recommendation system based on the multi-modal data of the wearable device includes an interest tracking module and an interest matching module. The interest tracking module inputs the individual multimedia information record into a preset preference weight learning model to obtain a multimedia information preference vector, inputs environment data and individual state data into a preset environment preference correction model to obtain a multimedia information preference correction vector, and further obtains an individual multimedia interest vector based on the multimedia information preference vector and the multimedia information preference correction vector; and then, the interest matching module obtains the individual multimedia interest characterization based on the multimedia interest vector and the historical multimedia information characterization, and performs cosine similarity matching on the individual multimedia interest characterization and the multimedia information characterization of the candidate multimedia information to obtain the recommendation probability of the candidate multimedia information. The following takes multimedia information as music as an example, and gives a specific implementation manner when the multimedia information recommendation system performs multimedia information recommendation:
S1: thirty pieces of music listening records of individual histories are obtained, an 11-dimension music representation is generated based on each piece of music record, and a 30×11-dimension history music representation R is obtained and expressed as follows;
s2: inputting a 30×11 historical music characterization representing a music listening history into a preset preference weight learning model, and outputting a music preference vector P of an individual according to a backbone network structure of a Resnet50, wherein the dimension is 30, and the model is expressed as follows;
P＝|P 1 ,P 2 ,...P 30 |
s3, combining weather, temperature, humidity, illumination, individual exercise amount, individual heart rate, individual PAI value and individual sleep condition within 30 days into 30 multiplied by 8 environment and individual state data;
s4, inputting 30 multiplied by 8 environment and individual state data into a preset environment preference correction model, and constructing the model according to a BiLSTM framework to obtain a music preference correction vector C, wherein the dimension is 30 and is expressed as follows;
C＝|C 1 ,C 2 ,...C 30 |
s5, synthesizing the individual music preference vector P and the music preference correction vector C into an individual music interest vector I in a point-to-point multiplication way, wherein the synthesis way is as follows;
I＝P×C＝|P 1 ×C 1 ,P 2 ×C 2 ,...P 30 ×C 30 |＝|I 1 ,I 2 ,...I 30 |
s6, multiplying the music interest vector I with the dimension of 30 by the historical music characterization R with the dimension of 30 multiplied by 11 to obtain the music interest characterization IR of the individual, wherein the dimension is 11, and the expression is as follows;
And S7, performing cosine similarity calculation on the individual music interest characterization IR and the characterization r of each candidate music, and normalizing the obtained matching degree D of each candidate music to obtain the recommendation probability D of each candidate music, wherein the recommendation probability D is expressed as follows.
Therefore, the multimedia information recommendation system based on the wearable device multi-mode data can obtain the individual multimedia interest characterization taking the individual multimedia information interest drift into consideration based on the environment data and the multi-mode data of the individual state data, and obtain the candidate multimedia information recommendation probability based on the individual multimedia interest characterization and the candidate multimedia information characterization, so that the change of the individual multimedia information preference is tracked, and the accuracy of the multimedia information recommendation is improved.
Referring to fig. 4, fig. 4 is a schematic diagram of a multimedia information recommendation device based on multi-mode data of a wearable device according to an embodiment of the present application, and as shown in fig. 4, the multimedia information recommendation device based on multi-mode data of a wearable device may include an obtaining module 401, a first processing module 402, a second processing module 403, a third processing module 404, a fourth processing module 405, and a recommendation module 406.
The acquiring module 401 is configured to acquire an individual multimedia information playing record, and acquire environmental data and individual state data acquired based on the wearable device; a first processing module 402, configured to generate a corresponding historical multimedia information representation according to the multimedia information play record; the second processing module 403 is configured to generate a corresponding multimedia information preference vector according to the historical multimedia information representation, and generate a corresponding multimedia information preference correction vector according to the state data and the environmental data; a third processing module 404, configured to generate an individual multimedia information interest vector according to the multimedia information preference vector and the multimedia information preference correction vector; a fourth processing module 405, configured to generate an individual multimedia information interest token according to the multimedia information interest vector and the historical multimedia information token; a recommendation module 406 for recommending multimedia information from a plurality of candidate multimedia information to an individual based on the multimedia information interest characterization.
In one implementation, the second processing module 403 is specifically configured to: inputting the historical multimedia information characterization into a preset multimedia information preference weight learning model to generate a multimedia information preference vector; the preference weight learning model is used for learning in advance to obtain the mapping relation between the historical multimedia information characterization and the individual multimedia information preference vector.
In an alternative implementation, the second processing module 403 is specifically configured to: inputting the state data and the environment data into a preset environment preference correction model to generate a multimedia information preference correction vector; the environment preference correction factor learning model is used for learning in advance to obtain state data and a mapping relation between the environment data and the multimedia information preference correction vector.
Optionally, cross stitch units are adopted to share information between the multimedia information preference weight learning model and the environment preference correction model.
In one implementation, the third processing module 404 is specifically configured to: and multiplying the multimedia information preference vector with the multimedia information preference correction vector point to generate an individual multimedia information interest vector.
In one implementation, recommendation module 406 is specifically configured to: acquiring multimedia information characterization of each candidate multimedia information; respectively carrying out similar matching on the multimedia information interest characterization and the multimedia information characterization of each candidate multimedia information to obtain the recommendation probability of each candidate multimedia information; selecting candidate multimedia information with recommendation probability meeting preset recommendation conditions from the plurality of candidate multimedia information; and recommending the candidate multimedia information with recommendation probability meeting the preset recommendation condition to the individual.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
By implementing the embodiment of the application, the change of the preference of the individual multimedia information can be tracked based on the individual multi-mode information (the environment information of the individual, the individual state information and the like), so that the accuracy of the recommendation of the multimedia information is improved.
Based on the embodiment of the application, the application also provides electronic equipment, which comprises: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the multimedia information recommendation method based on the wearable device multimodal data of any of the foregoing embodiments.
Based on the embodiment of the application, the application also provides a computer readable storage medium, wherein, the computer instructions are used for making a computer execute the multimedia information recommendation method based on the multi-mode data of the wearable device according to any of the previous embodiments provided by the embodiment of the application.
Referring now to fig. 5, shown in fig. 5 is a schematic block diagram of an example electronic device that may be used to implement an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.
As shown in fig. 5, the apparatus 500 includes a computing unit 501 that can perform various appropriate actions and processes according to a computer program stored in a Read-Only Memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access Memory (Random Access Memory, RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, ROM 502, and RAM 503 are connected to each other by a bus 504. An Input/Output (I/O) interface 505 is also connected to bus 504.
Various components in the device 500 are connected to the I/O interface 505, including: an input unit 506 such as a keyboard, a mouse, etc.; an output unit 507 such as various types of displays, speakers, and the like; a storage unit 508 such as a magnetic disk, an optical disk, or the like; and a communication unit 509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 501 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 501 include, but are not limited to, a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (Digital Signal Process, DSP), and any suitable processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a multimedia information recommendation method based on wearable device multimodal data. For example, in some embodiments, the multimedia information recommendation method based on wearable device multimodal data may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 500 via the ROM 502 and/or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the above-described multimedia information recommendation method based on wearable device multimodal data may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the multimedia information recommendation method based on the wearable device multimodal data in any other suitable way (e.g. by means of firmware).
Various implementations of the systems and techniques described here above can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (Field Programmable Gate Array, FPGAs), application specific integrated circuits (Application SpecificIntegrated Circuit, ASICs), application specific standard products (Application Specific Standard Parts, ASSPs), systems On Chip (SOC), load programmable logic devices (Complex Programmable Logic Device, CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present application may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (EPROM) or flash Memory, an optical fiber, a portable compact disc read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., cathode Ray Tube (CRT) or LCD (Liquid Crystal Display ) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (Local Area Network, LAN), wide area network (Wide Area Network, WAN), the internet and blockchain networks.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS (Virtual Private Server virtual special server) service are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solution of the present application are achieved, and are not limited herein.
The above embodiments do not limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of the present application.
Claims (15)
1. The multimedia information recommendation method based on the multi-mode data of the wearable device is characterized by comprising the following steps of:
acquiring individual multimedia information playing records, and acquiring environmental data acquired based on the wearable equipment and state data of the individual;
generating a corresponding historical multimedia information representation according to the multimedia information playing record;
generating a corresponding multimedia information preference vector according to the historical multimedia information representation, and generating a corresponding multimedia information preference correction vector according to the state data and the environment data;
Generating a multimedia information interest vector of the individual according to the multimedia information preference vector and the multimedia information preference correction vector;
generating a multimedia information interest representation of the individual according to the multimedia information interest vector and the historical multimedia information representation;
based on the multimedia information interest characterization, multimedia information is recommended from a plurality of candidate multimedia information to the individual.
2. The method of claim 1, wherein generating the corresponding multimedia information preference vector from the historical multimedia information characterization comprises:
inputting the historical multimedia information characterization into a preset multimedia information preference weight learning model to generate the multimedia information preference vector;
wherein the preference weight learning model has learned in advance to obtain a mapping relationship between the historical multimedia information characterization and the multimedia information preference vector.
3. The method of claim 2, wherein generating the corresponding multimedia information preference correction vector from the status data and the environmental data comprises:
inputting the state data and the environment data into a preset environment preference correction model to generate the multimedia information preference correction vector;
Wherein the environment preference correction factor learning model has learned in advance to obtain the state data and a mapping relationship between the environment data and the multimedia information preference correction vector.
4. The method of claim 3, wherein a cross stitch unit is used to share information between the multimedia information preference weight learning model and the environmental preference correction model.
5. The method of claim 1, wherein the generating the individual multimedia information interest vector from the multimedia information preference vector and the multimedia information preference correction vector comprises:
and multiplying the multimedia information preference vector with the multimedia information preference correction vector point to generate the individual multimedia information interest vector.
6. The method of claim 1, wherein recommending multimedia information to the individual from a plurality of candidate multimedia information based on the multimedia information interest characterization comprises:
acquiring multimedia information characterization of each candidate multimedia information;
respectively carrying out similar matching on the multimedia information interest characterization and the multimedia information characterization of each candidate multimedia information to obtain the recommendation probability of each candidate multimedia information;
Selecting candidate multimedia information with the recommendation probability meeting a preset recommendation condition from the plurality of candidate multimedia information;
and recommending the candidate multimedia information with the recommendation probability meeting the preset recommendation condition to the individual.
7. A multimedia information recommendation device based on multi-modal data of a wearable device, comprising:
the acquisition module is used for acquiring individual multimedia information playing records and acquiring environmental data acquired based on the wearable equipment and state data of the individual;
the first processing module is used for generating a corresponding historical multimedia information representation according to the multimedia information playing record;
the second processing module is used for generating a corresponding multimedia information preference vector according to the historical multimedia information representation and generating a corresponding multimedia information preference correction vector according to the state data and the environment data;
a third processing module for generating a multimedia information interest vector of the individual according to the multimedia information preference vector and the multimedia information preference correction vector
A fourth processing module, configured to generate a multimedia information interest representation of the individual according to the multimedia information interest vector and the historical multimedia information representation;
And the recommendation module is used for recommending the multimedia information to the individual from the plurality of candidate multimedia information based on the multimedia information interest characterization.
8. The apparatus of claim 7, wherein the second processing module is specifically configured to:
inputting the historical multimedia information characterization into a preset multimedia information preference weight learning model to generate the multimedia information preference vector;
wherein the preference weight learning model has learned in advance to obtain a mapping relationship between the historical multimedia information characterization and the multimedia information preference vector.
9. The apparatus of claim 8, wherein the second processing module is specifically configured to:
inputting the state data and the environment data into a preset environment preference correction model to generate the multimedia information preference correction vector;
wherein the environment preference correction factor learning model has learned in advance to obtain the state data and a mapping relationship between the environment data and the multimedia information preference correction vector.
10. The apparatus of claim 9, wherein a cross-stitch unit is used to share information between the multimedia information preference weight learning model and the environmental preference correction model.
11. The apparatus of claim 7, wherein the third processing module is specifically configured to:
and multiplying the multimedia information preference vector with the multimedia information preference correction vector point to generate the individual multimedia information interest vector.
12. The apparatus of claim 7, wherein the recommendation module is specifically configured to:
acquiring multimedia information characterization of each candidate multimedia information;
respectively carrying out similar matching on the multimedia information interest characterization and the multimedia information characterization of each candidate multimedia information to obtain the recommendation probability of each candidate multimedia information;
selecting candidate multimedia information with the recommendation probability meeting a preset recommendation condition from the plurality of candidate multimedia information;
and recommending the candidate multimedia information with the recommendation probability meeting the preset recommendation condition to the individual.
13. An electronic device, comprising:
at least one processor;
and a memory communicatively coupled to the at least one processor;
wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the wearable device multimodal data-based multimedia information recommendation method of any of claims 1 to 6.
14. A computer readable storage medium storing instructions which, when executed, cause the method of any one of claims 1 to 6 to be implemented.
15. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the wearable device multimodal data-based multimedia information recommendation method of any one of claims 1 to 6.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210306451.2A CN116842202A (en) | 2022-03-25 | 2022-03-25 | Multimedia information recommendation method and device based on multi-mode data of wearable equipment |
| PCT/CN2023/083692 WO2023179765A1 (en) | 2022-03-25 | 2023-03-24 | Multimedia recommendation method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210306451.2A CN116842202A (en) | 2022-03-25 | 2022-03-25 | Multimedia information recommendation method and device based on multi-mode data of wearable equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116842202A true CN116842202A (en) | 2023-10-03 |
Family
ID=88158620
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210306451.2A Pending CN116842202A (en) | 2022-03-25 | 2022-03-25 | Multimedia information recommendation method and device based on multi-mode data of wearable equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116842202A (en) |
-
2022
- 2022-03-25 CN CN202210306451.2A patent/CN116842202A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109902706B (en) | Recommendation method and device | |
| CN112289299B (en) | Training method and device of speech synthesis model, storage medium and electronic equipment | |
| EP3493032A1 (en) | Robot control method and companion robot | |
| CN112365876B (en) | Method, device and equipment for training speech synthesis model and storage medium | |
| CN116072098B (en) | Audio signal generation method, model training method, device, equipment and medium | |
| CN112614478B (en) | Audio training data processing method, device, equipment and storage medium | |
| CN113301442A (en) | Method, apparatus, medium, and program product for determining live broadcast resource | |
| CN111104964B (en) | Method, equipment and computer storage medium for matching music with action | |
| WO2015153240A1 (en) | Directed recommendations | |
| CN114490926A (en) | Method and device for determining similar problems, storage medium and terminal | |
| CN117152308B (en) | Virtual person action expression optimization method and system | |
| CN111445922B (en) | Audio matching method, device, computer equipment and storage medium | |
| CN116821475B (en) | Video recommendation method and device based on client data and computer equipment | |
| CN112949433A (en) | Method, device and equipment for generating video classification model and storage medium | |
| CN116910201A (en) | Dialogue data generation method and related equipment thereof | |
| WO2023061229A1 (en) | Video generation method and device | |
| CN116842202A (en) | Multimedia information recommendation method and device based on multi-mode data of wearable equipment | |
| CN114676227B (en) | Sample generation method, model training method and retrieval method | |
| CN110659382B (en) | Mixed music recommendation method based on heterogeneous information network representation learning technology | |
| CN112562641A (en) | Method, device, equipment and storage medium for evaluating voice interaction satisfaction degree | |
| CN113886639A (en) | Digital human video generation method and device, electronic equipment and storage medium | |
| CN110688529A (en) | Method and device for retrieving video and electronic equipment | |
| CN115292467B (en) | Information processing and model training method, device, equipment, medium and program product | |
| TWI817921B (en) | Model modeling instruction generation method and system | |
| WO2023231596A1 (en) | Voice conversion model training method and apparatus, and voice conversion method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |