CN112329604A - Multi-modal emotion analysis method based on multi-dimensional low-rank decomposition - Google Patents

Multi-modal emotion analysis method based on multi-dimensional low-rank decomposition Download PDF

Info

Publication number
CN112329604A
CN112329604A CN202011209001.9A CN202011209001A CN112329604A CN 112329604 A CN112329604 A CN 112329604A CN 202011209001 A CN202011209001 A CN 202011209001A CN 112329604 A CN112329604 A CN 112329604A
Authority
CN
China
Prior art keywords
video
modal
features
rank decomposition
dimensional low
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011209001.9A
Other languages
Chinese (zh)
Other versions
CN112329604B (en
Inventor
金涛
李英明
张仲非
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Priority to CN202011209001.9A priority Critical patent/CN112329604B/en
Publication of CN112329604A publication Critical patent/CN112329604A/en
Application granted granted Critical
Publication of CN112329604B publication Critical patent/CN112329604B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/284—Lexical analysis, e.g. tokenisation or collocates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Multimedia (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Acoustics & Sound (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Child & Adolescent Psychology (AREA)
  • Hospice & Palliative Care (AREA)
  • Psychiatry (AREA)
  • Image Analysis (AREA)

Abstract

本发明公开了一种基于多维度低秩分解的多模态情感分析方法，用于将高维度的多模态特征融合成为低维的向量，再用于视频情感分析。具体包括如下步骤：获取用于训练多模态情感分析模型的视频数据集，所述视频数据集包括多个样本视频，并定义算法目标；对视频数据集中的图像特征，音频特征以及文本特征进行提取，得到视频数据的图像特征，音频特征以及文本特征；基于提取的图像特征，音频特征以及文本特征建立基于多维度低秩分解机制的多模态情感分析模型；使用所述多模态情感分析模型对输入视频进行情感分析。本发明适用于真实视频场景的多模态情感分析，面对各类复杂情况具有较佳的效果和鲁棒性。

Figure 202011209001

The invention discloses a multi-modal sentiment analysis method based on multi-dimensional low-rank decomposition, which is used to fuse high-dimensional multi-modal features into low-dimensional vectors, which are then used for video sentiment analysis. Specifically, it includes the following steps: obtaining a video data set for training a multimodal sentiment analysis model, the video data set includes a plurality of sample videos, and defining algorithm targets; image features, audio features and text features in the video data set are analyzed Extracting and obtaining image features, audio features and text features of the video data; establishing a multi-modal sentiment analysis model based on a multi-dimensional low-rank decomposition mechanism based on the extracted image features, audio features and text features; using the multi-modal sentiment analysis The model performs sentiment analysis on the input video. The invention is suitable for multi-modal emotion analysis of real video scenes, and has better effect and robustness in the face of various complex situations.

Figure 202011209001

Description

Multi-modal emotion analysis method based on multi-dimensional low-rank decomposition
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a multi-modal emotion analysis method based on multi-dimensional low-rank decomposition.
Background
In modern society, video becomes an indispensable part of human society, and is said to be ubiquitous. Such environments have led to significant developments in the research of semantic content of video. The multi-modal emotion analysis is a relatively important branch of video analysis, and is particularly important in the era of rapid development of short video live broadcast in the modern society. The method can judge the change of emotion of the video speaker in real time according to the expression, language and sound of the video speaker, and serve subsequent applications.
Most of the existing multi-modal emotion analysis methods based on tensors adopt the method that feature mean values of different modes are pooled, and then features are mapped to high-order tensor representations to be applied to subsequent tasks. Such processing methods ignore rich video timing information.
Disclosure of Invention
In order to solve the above problems, the present invention provides a multi-modal emotion analysis method based on multi-dimensional low-rank decomposition, which is used for predicting the emotion score of a speaker in a video. The method first extracts the multi-modal features of the video including images, audio, and text. And then, feature fusion is carried out on multi-modal feature multi-dimension by using a tensor low-rank approximation method, so that the complexity of the model is reduced. The method makes full use of various modes in the video data, overcomes the defect that the existing tensor method ignores the time sequence information, and has better expandability.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a multi-modal emotion analysis method based on multi-dimensional low-rank decomposition comprises the following steps:
s1, acquiring a video data set for training a multi-modal emotion analysis model, wherein the video data set comprises a plurality of sample videos and defines an algorithm target;
s2, extracting image features, audio features and text features in the video data set to obtain the image features, the audio features and the text features of the video data;
s3, establishing a multi-modal emotion analysis model based on the multi-dimensional low-rank decomposition mechanism based on the extracted image features, audio features and text features;
and S4, performing emotion analysis on the input video by using the multi-modal emotion analysis model.
Further, in step S1, the video data set includes a video XtrainAnd manually labeled sentiment score Ytrain；
The algorithm targets are defined as: given video x ═ x1，x2，...，xL}，xiRepresenting the ith block, each video block containing a fixed number of video frames, L representing the total number of video blocks, and the sentiment score y for the predicted video segment, y being a continuous value score.
Further, step S2 specifically includes:
s21, taking each video block xiAll the images in (1) are input into a two-dimensional convolution neural network, the image characteristics of the video are extracted, and the mean vector is calculated and recorded as
Figure BDA0002758010740000021
S22, extracting each video block xiThe text in (1) is characterized by word vectors, and the mean vector is calculated and recorded as
Figure BDA0002758010740000022
S23, extracting traditional MFCC audio features based on video blocks
Figure BDA0002758010740000023
S24, obtaining all video block image characteristics based on the extraction result
Figure BDA0002758010740000024
Text features
Figure BDA00027580107400000213
Figure BDA0002758010740000026
Audio features
Figure BDA0002758010740000027
Further, in step S3, the multi-modal emotion analysis model based on multi-dimensional low-rank decomposition is composed of a series of linear layers, dot product layers and mean pooling layers, and its video representation
Figure BDA0002758010740000028
Calculated by the following formula:
Figure BDA0002758010740000029
wherein, VmRepresenting modal characteristics including image S, audio A or text T, and
Figure BDA00027580107400000210
and
Figure BDA00027580107400000211
is a training parameter, wherein R1And R2The rank of the expression tensor is manually set;
the emotion scores of the speakers in the video are predicted based on the video representation o,
p＝Woo+bo
wherein
Figure BDA00027580107400000212
bo∈R1Is a training parameter and p represents the predicted emotion score.
Further, the multi-modal emotion analysis model training uses an L1 loss function of the predicted value p and the label value y,
Loss＝|y-p|1
where the entire model is trained under the Loss function Loss using Adam optimization algorithm and back propagation algorithm.
Compared with the existing multi-modal emotion analysis method, the multi-modal emotion analysis method based on multi-dimensional low-rank decomposition has the following beneficial effects:
firstly, the time sequence characteristics are introduced into a tensor fusion method, and the defects of the existing method are greatly supplemented.
Secondly, the invention firstly proposes tensor fusion on a plurality of dimensions, and a method for carrying out low-rank decomposition approximation on the plurality of dimensions is proposed through deduction, so that the performance of the model is improved, and the efficiency is not lost.
The method has good application value in short video and live broadcast systems, and can effectively improve the accuracy of multi-modal emotion analysis.
Drawings
FIG. 1 is a flow diagram of a multi-modal sentiment analysis method based on multi-dimensional low-rank decomposition according to the present invention.
FIG. 2 is a frame diagram of a multi-modal sentiment analysis model based on multi-dimensional low-rank decomposition according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
On the contrary, the invention is intended to cover alternatives, modifications, equivalents and alternatives which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, certain specific details are set forth in order to provide a better understanding of the present invention. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details.
Referring to fig. 1, in a preferred embodiment of the present invention, a multi-modal sentiment analysis method based on multi-dimensional low rank decomposition includes the following steps:
first, a video data set for training a multimodal emotion analysis model is acquired. Wherein the video data set used for training the emotion analysis model comprises a video XtrainArtificially labeled video description sentence Ytrain；
The algorithm targets are defined as: given video x ═ x1，x2，...，xL}，xiRepresenting the ith block, each video block containing a fixed number of video frames, L representing the total number of video blocks, and the sentiment score y for the predicted video segment, y being a continuous value score.
Second, multi-modal features in the video data set are extracted. Specifically, the method comprises the following steps:
first, take each video block xiAll the images in (1) are input into a two-dimensional convolution neural network, the image characteristics of the video are extracted, and the mean vector is calculated and recorded as
Figure BDA0002758010740000041
Second, extract each video block xiThe text in (1) is characterized by word vectors, and the mean vector is calculated and recorded as
Figure BDA0002758010740000042
Thirdly, extracting traditional MFCC audio features based on video blocks and recording the traditional MFCC audio features as
Figure BDA0002758010740000043
Fourthly, obtaining all video block image characteristics based on the extraction result
Figure BDA0002758010740000044
Text features
Figure BDA0002758010740000045
Audio features
Figure BDA0002758010740000046
And then, establishing a multi-modal emotion analysis model based on a multi-dimensional low-rank decomposition mechanism based on the extracted image features, audio features and text features.
The multi-modal emotion analysis model based on multi-dimensional low-rank decomposition is composed of a series of linear layers, a dot product layer and a mean pooling layer, and video representation of the model
Figure BDA00027580107400000412
Calculated by the following formula:
Figure BDA0002758010740000048
wherein, VmRepresenting modal characteristics including image S, audio A or text T, and
Figure BDA0002758010740000049
and
Figure BDA00027580107400000410
is a training parameter, wherein R1And R2The rank of the expression tensor is manually set;
the emotion scores of the speakers in the video are predicted based on the video representation o described above,
p＝Woo+bo
wherein
Figure BDA00027580107400000411
bo∈R1Is a training parameter, p represents the predicted sentiment score;
further, the training of the multi-model emotion analysis model uses an L1 loss function of the predicted value p and the label value y,
Loss＝|y-p|1
where the entire model is trained under the Loss function Loss using Adam optimization algorithm and back propagation algorithm.
In the above embodiments, the multi-modal emotion analysis method of the present invention uses image features, audio features and text features in the video. On the basis, a multi-dimensional low-rank decomposition mechanism is established. And finally, performing emotion analysis on the unmarked video by using the trained model.
Through the technical scheme, the embodiment of the invention develops a multi-modal emotion analysis method algorithm applied to the unprocessed video based on the deep learning technology. The time sequence information is introduced into the existing tensor fusion method, the tensor fusion method is used in multiple dimensions, and the model efficiency is improved by low-rank decomposition, so that the emotion analysis is more accurate and rapid.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims (5)

1. A multi-modal emotion analysis method based on multi-dimensional low-rank decomposition is characterized by comprising the following steps:
s1, acquiring a video data set for training a multi-modal emotion analysis model, wherein the video data set comprises a plurality of sample videos and defines an algorithm target;
s2, extracting image features, audio features and text features in the video data set to obtain the image features, the audio features and the text features of the video data;
s3, establishing a multi-modal emotion analysis model based on the multi-dimensional low-rank decomposition mechanism based on the extracted image features, audio features and text features;
and S4, performing emotion analysis on the input video by using the multi-modal emotion analysis model.
2. The method for multi-modal sentiment analysis based on multi-dimensional low rank decomposition of claim 1 wherein in step S1, the video data set comprises video XtrainAnd manually labeled sentiment score Ytrain；
The algorithm targets are defined as:given video x ═ x1，x2，...，xL}，xiRepresenting the ith block, each video block containing a fixed number of video frames, L representing the total number of video blocks, and the sentiment score y for the predicted video segment, y being a continuous value score.
3. The multi-modal sentiment analysis method based on multi-dimensional low rank decomposition according to claim 2, wherein the step S2 specifically comprises:
s21, taking each video block xiAll the images in (1) are input into a two-dimensional convolution neural network, the image characteristics of the video are extracted, and the mean vector is calculated and recorded as
Figure FDA0002758010730000011
S22, extracting each video block xiThe text in (1) is characterized by word vectors, and the mean vector is calculated and recorded as
Figure FDA0002758010730000012
S23, extracting traditional MFCC audio features based on video blocks
Figure FDA0002758010730000013
S24, obtaining all video block image characteristics based on the extraction result
Figure FDA0002758010730000014
Text features
Figure FDA0002758010730000015
Figure FDA0002758010730000016
Audio features
Figure FDA0002758010730000017
4. The method as claimed in claim 3, wherein in step S3, the multi-modal sentiment analysis model based on multi-dimensional low-rank decomposition is composed of a series of linear layers, dot product layers and mean pooling layers, and its video representation is
Figure FDA0002758010730000021
Calculated by the following formula:
Figure FDA0002758010730000022
wherein, VmRepresenting modal characteristics including image S, audio A or text T, and
Figure FDA0002758010730000023
and
Figure FDA0002758010730000024
is a training parameter, wherein R1And R2The rank of the expression tensor is manually set; based on the above video characterizationoThe emotion scores of the speakers in the video are predicted,
p＝Woo+bo
wherein
Figure FDA0002758010730000025
Is a training parameter and p represents the predicted emotion score.
5. The multi-dimensional low-rank decomposition based multi-modal sentiment analysis method of claim 4, wherein the training of the multi-modal sentiment analysis model uses an L1 loss function of a predicted value p and a tag value y.
Loss＝|y-p|1
Where the entire model is trained under the Loss function Loss using Adam optimization algorithm and back propagation algorithm.
CN202011209001.9A 2020-11-03 2020-11-03 A Multimodal Sentiment Analysis Method Based on Multidimensional Low-Rank Decomposition Active CN112329604B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011209001.9A CN112329604B (en) 2020-11-03 2020-11-03 A Multimodal Sentiment Analysis Method Based on Multidimensional Low-Rank Decomposition

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011209001.9A CN112329604B (en) 2020-11-03 2020-11-03 A Multimodal Sentiment Analysis Method Based on Multidimensional Low-Rank Decomposition

Publications (2)

Publication Number Publication Date
CN112329604A true CN112329604A (en) 2021-02-05
CN112329604B CN112329604B (en) 2022-09-20

Family

ID=74322845

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011209001.9A Active CN112329604B (en) 2020-11-03 2020-11-03 A Multimodal Sentiment Analysis Method Based on Multidimensional Low-Rank Decomposition

Country Status (1)

Country Link
CN (1) CN112329604B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114999006A (en) * 2022-05-20 2022-09-02 南京邮电大学 Method, Apparatus and Equipment for Multimodal Sentiment Analysis Based on Uncertainty Estimation
US20230214450A1 (en) * 2021-12-31 2023-07-06 Dell Products L.P. Method, electronic device, and computer program product for training model
WO2023163383A1 (en) * 2022-02-28 2023-08-31 에스케이텔레콤 주식회사 Multimodal-based method and apparatus for recognizing emotion in real time
CN117688936A (en) * 2024-02-04 2024-03-12 江西农业大学 Low-rank multi-mode fusion emotion analysis method for graphic fusion
CN119028380A (en) * 2024-10-25 2024-11-26 山东财经大学 Online business conversation record annotation method and system based on NLP

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103310229A (en) * 2013-06-15 2013-09-18 浙江大学 Multitask machine learning method and multitask machine learning device both used for image classification
CN104299216A (en) * 2014-10-22 2015-01-21 北京航空航天大学 Multimodality medical image fusion method based on multiscale anisotropic decomposition and low rank analysis
CN106056082A (en) * 2016-05-31 2016-10-26 杭州电子科技大学 Video action recognition method based on sparse low-rank coding
CN107292858A (en) * 2017-05-22 2017-10-24 昆明理工大学 A kind of multimode medical image fusion method based on low-rank decomposition and rarefaction representation
CN108197629A (en) * 2017-12-30 2018-06-22 北京工业大学 A kind of Multimodal medical image feature extracting method based on label correlation constraint tensor resolution
CN109934135A (en) * 2019-02-28 2019-06-25 北京航空航天大学 A low-rank matrix factorization-based method for detecting foreign objects in railway tracks
CN110188770A (en) * 2019-05-17 2019-08-30 重庆邮电大学 A non-convex low-rank salient object detection method based on structural matrix factorization
CN110222213A (en) * 2019-05-28 2019-09-10 天津大学 A kind of image classification method based on isomery tensor resolution
CN111178389A (en) * 2019-12-06 2020-05-19 杭州电子科技大学 Multi-modal deep hierarchical fusion sentiment analysis method based on multi-channel tensor pooling

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103310229A (en) * 2013-06-15 2013-09-18 浙江大学 Multitask machine learning method and multitask machine learning device both used for image classification
CN104299216A (en) * 2014-10-22 2015-01-21 北京航空航天大学 Multimodality medical image fusion method based on multiscale anisotropic decomposition and low rank analysis
CN106056082A (en) * 2016-05-31 2016-10-26 杭州电子科技大学 Video action recognition method based on sparse low-rank coding
CN107292858A (en) * 2017-05-22 2017-10-24 昆明理工大学 A kind of multimode medical image fusion method based on low-rank decomposition and rarefaction representation
CN108197629A (en) * 2017-12-30 2018-06-22 北京工业大学 A kind of Multimodal medical image feature extracting method based on label correlation constraint tensor resolution
CN109934135A (en) * 2019-02-28 2019-06-25 北京航空航天大学 A low-rank matrix factorization-based method for detecting foreign objects in railway tracks
CN110188770A (en) * 2019-05-17 2019-08-30 重庆邮电大学 A non-convex low-rank salient object detection method based on structural matrix factorization
CN110222213A (en) * 2019-05-28 2019-09-10 天津大学 A kind of image classification method based on isomery tensor resolution
CN111178389A (en) * 2019-12-06 2020-05-19 杭州电子科技大学 Multi-modal deep hierarchical fusion sentiment analysis method based on multi-channel tensor pooling

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ZHUN LIU 等: "Efficient Low-rank Multimodal Fusion with Modality-Specific Factors", 《ARXIV:1806.00064V1 [CS.AI]》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230214450A1 (en) * 2021-12-31 2023-07-06 Dell Products L.P. Method, electronic device, and computer program product for training model
WO2023163383A1 (en) * 2022-02-28 2023-08-31 에스케이텔레콤 주식회사 Multimodal-based method and apparatus for recognizing emotion in real time
CN114999006A (en) * 2022-05-20 2022-09-02 南京邮电大学 Method, Apparatus and Equipment for Multimodal Sentiment Analysis Based on Uncertainty Estimation
CN114999006B (en) * 2022-05-20 2024-11-15 南京邮电大学 Multimodal sentiment analysis method, device and equipment based on uncertainty estimation
CN117688936A (en) * 2024-02-04 2024-03-12 江西农业大学 Low-rank multi-mode fusion emotion analysis method for graphic fusion
CN117688936B (en) * 2024-02-04 2024-04-19 江西农业大学 A low-rank multimodal fusion sentiment analysis method based on image and text fusion
CN119028380A (en) * 2024-10-25 2024-11-26 山东财经大学 Online business conversation record annotation method and system based on NLP

Also Published As

Publication number Publication date
CN112329604B (en) 2022-09-20

Similar Documents

Publication Publication Date Title
CN112329604A (en) Multi-modal emotion analysis method based on multi-dimensional low-rank decomposition
WO2023065617A1 (en) Cross-modal retrieval system and method based on pre-training model and recall and ranking
CN110532996B (en) Video classification method, information processing method and server
CN111931795B (en) Multimodal emotion recognition method and system based on subspace sparse feature fusion
WO2020108396A1 (en) Video classification method, and server
CN114282059B (en) Video retrieval method, device, equipment and storage medium
WO2024235271A1 (en) Movement generation method and apparatus for virtual character, and construction method and apparatus for movement library of virtual avatar
CN117689963A (en) A visual entity linking method based on multi-modal pre-training model
CN110866129A (en) Cross-media retrieval method based on cross-media uniform characterization model
CN113220865A (en) Text similar vocabulary retrieval method, system, medium and electronic equipment
CN116226347A (en) A method and system for fine-grained video emotional content question answering based on multi-modal data
CN116701996A (en) Multi-modal emotion analysis method, system, equipment and medium based on multiple loss functions
CN116167014A (en) Multi-mode associated emotion recognition method and system based on vision and voice
CN114741556B (en) Short video classification method based on scene segment and multi-mode feature enhancement
WO2023134085A1 (en) Question answer prediction method and prediction apparatus, electronic device, and storage medium
CN113053361B (en) Speech recognition method, model training method, device, equipment and medium
CN113823271B (en) Training method and device for voice classification model, computer equipment and storage medium
CN117540007B (en) Multi-mode emotion analysis method, system and equipment based on similar mode completion
CN114048314A (en) A Natural Language Steganalysis Method
CN117152851B (en) Face and human body collaborative clustering method based on large model pre-training
CN117370934B (en) A multimodal data enhancement method for sensitive information discovery models
CN111209433A (en) Video classification algorithm based on feature enhancement
CN117668262A (en) Sound image file utilization system based on artificial intelligent voice and image recognition technology
CN115017900B (en) A conversation emotion recognition method based on multimodal multi-debiasing
CN110674265A (en) Unstructured information oriented feature discrimination and information recommendation system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant