CN114662497A - False news detection method based on cooperative neural network - Google Patents

False news detection method based on cooperative neural network Download PDF

Info

Publication number
CN114662497A
CN114662497A CN202210173809.9A CN202210173809A CN114662497A CN 114662497 A CN114662497 A CN 114662497A CN 202210173809 A CN202210173809 A CN 202210173809A CN 114662497 A CN114662497 A CN 114662497A
Authority
CN
China
Prior art keywords
visual
text
extraction module
features
feature extraction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210173809.9A
Other languages
Chinese (zh)
Inventor
薛均晓
翟蓝航
石磊
高宇飞
刘成明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhengzhou University
Original Assignee
Zhengzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhengzhou University filed Critical Zhengzhou University
Priority to CN202210173809.9A priority Critical patent/CN114662497A/en
Publication of CN114662497A publication Critical patent/CN114662497A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/211—Selection of the most significant subset of features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention is suitable for the technical field of computer vision and graphic image processing, and provides a false news detection method based on a collaborative neural network, which comprises a text feature extraction module, a visual semantic feature extraction module, a visual tampering feature extraction module, a similarity measurement module and a multi-mode fusion module, and further comprises the following steps: step S1: the text feature extraction module and the visual semantic feature extraction module are responsible for extracting text and visual semantic features and mapping the text and the visual semantic features to the same space; step S2: the visual tampering feature extraction module is responsible for extracting visual physical features and tampering features; through the cooperation of the five sub-networks, the similarity of different modal data in the multi-modal news data, semantic hierarchical features of texts and images and some physical hierarchical features of visual patterns are well captured, so that the false news detection in complex scenes is more suitable than the existing models.

Description

False news detection method based on cooperative neural network
Technical Field
The invention belongs to the technical field of computer vision and graphic image processing, and particularly relates to a false news detection method based on a collaborative neural network.
Background
The method based on machine learning comprises the steps of firstly extracting emotion polarity, user influence and geographic positions by utilizing feature engineering, and then classifying events into fake news and real news by utilizing classifiers such as a training decision tree and a support vector machine. And then, adopting a decision tree to detect rumors according to the characteristics of emotional scores, the number of websites on the microblog, the number of days of user registration and the like.
The method can improve the detection precision of false news, but ignores multi-modal data features of news serving as a multi-modal data set, so that text and visual information of the false news cannot be effectively utilized.
A multimodal approach based on deep learning that utilizes an image-text consistency driven multimodal approach to analyze social media sentiment, a new attention recurrent neural network and dynamically interpretable recommendations for visual fusion. The method mainly solves the problem of how to integrate different forms of information, but because the model adopts a pre-trained image description generation model, the similarity of multi-modal data cannot be directly calculated, and thus the use of scenes is greatly limited.
The method is used for detecting the similarity of false news pictures and texts in false news detection and is used as a part of false news identification characteristics in multi-modal data.
And the branch network is designed to better extract the visual semantic vector to obtain better image semantic expression so as to better capture the semantic features of the fake news in the visual expression.
The two technologies are an error level analysis algorithm and a convolutional neural network, and the authenticity of the news pictures can be judged better on a physical level.
Disclosure of Invention
The invention provides a false news detection method based on a collaborative neural network, and aims to solve the problems.
The invention is realized in this way, a false news detection method based on the cooperative neural network, including text characteristic extraction module, visual semantic characteristic extraction module, visual tampering characteristic extraction module, similarity measurement module and multimodal fusion module, also includes the following steps:
step S1: the text feature extraction module and the visual semantic feature extraction module are responsible for extracting text and visual semantic features and mapping the text and the visual semantic features to the same space;
step S2: the visual tampering feature extraction module is responsible for extracting visual physical features and tampering features;
step S3: the similarity measurement module can directly measure the similarity of the news multi-mode data aiming at the problem that the image is not matched with the text.
Preferably, in the text feature extraction module, a BERT pre-training model is used to extract text features, BiGRU is used to extract the BERT extracted features, BiGRU is used to extract time attributes of the text features, and the text features are converted into a text feature sequence.
Preferably, in the visual semantic feature extraction module, the output of the convolutional neural network is used as a low-level feature of the image, and then is fused with the tamper detection part.
Preferably, the input image is encoded by a ResNet50 pre-trained model, and the image features are encoded using a 1024-dimensional fully-connected layer before the classification layer of the pre-trained ResNet50 model.
Preferably, in the visual tampering feature extraction module, the image is processed through visual transformation or tampering, and the ResNet50 model is applied to extract the image tampering feature.
Preferably, in the similarity measurement module, vector representations of the text and the image are obtained through a visual semantic feature extraction module and a text feature extraction module.
Preferably, in the multi-modal fusion module, the fusion features of the image and the text are obtained through the text feature extraction module, the visual semantic feature extraction module, the visual tampering feature extraction module and the similarity measurement module, and attention weight is given.
Preferably, in the multi-modal fusion module, an attention mechanism is used to assign weighted image features and semantic hierarchy features of images and texts to the physical hierarchy.
Compared with the prior art, the invention has the beneficial effects that: the false news detection method based on the cooperative neural network well captures the similarity of different modal data in multi-modal news data, semantic hierarchical features of texts and images and some physical hierarchical features of visual patterns through the common work of five sub-networks (a text feature extraction module, a visual semantic feature extraction module, a visual tampering feature extraction module, a similarity measurement module and a multi-modal fusion module), so that the false news detection in a complex scene is more suitable than the existing model.
Drawings
FIG. 1 is a schematic diagram of the process steps of the present invention;
FIG. 2 is a schematic diagram of a multi-modal data collaborative neural network architecture in accordance with the present invention;
FIG. 3 is a schematic diagram of a comparison diagram of the states before and after processing by an error level analysis algorithm in the present invention;
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Referring to fig. 1-3, the present invention provides a technical solution: a false news detection method based on a collaborative neural network comprises a text feature extraction module, a visual semantic feature extraction module, a visual tampering feature extraction module, a similarity measurement module and a multi-mode fusion module, and further comprises the following steps:
step S1: the text feature extraction module and the visual semantic feature extraction module are responsible for extracting text and visual semantic features and mapping the text and the visual semantic features to the same space;
step S2: the visual tampering feature extraction module is responsible for extracting visual physical features and tampering features;
step S3: the similarity measurement module can directly measure the similarity of the news multi-mode data aiming at the problem that the image is not matched with the text.
As shown in fig. 2, the architecture of the cooperative neural network is shown, where a blue network is a text feature extraction module, a red network is a visual semantic feature extraction module, an orange network is a visual tampering feature extraction module, a purple network is a similarity measurement module, and a pink network is a multi-modal fusion module, and the implementation methods of the modules are as follows:
in the present embodiment, in the text feature extraction module, it is not so simple for the sentences that need to be processed in reality, for example, the context of a word should be considered for solving the word ambiguity problem, and for this problem, the text feature is extracted by using the BERT pre-training model, as shown in equation (1):
Figure BDA0003519652010000041
wherein, tiRepresenting the input text sequence, ht iRepresenting the text feature vector after the BERT was embedded. Then, in order to better capture global feature information and achieve better integration with image semantic information, BiGRU (BiGRU is a neural network model consisting of unidirectional, opposite-direction GRU whose output is determined jointly by the states of two GRUs, at each instant the input will simultaneously provide two opposite-direction GRU's, and the output is determined jointly by the two unidirectional GRUs) is used to extract BERT-extracted features, which BiGRU can further extract temporal attributes of text features, converting the text features into a text feature sequence. As shown in equation (2):
Figure BDA0003519652010000042
wherein h ist iRepresenting the text feature vector after embedding the BERT, fi tRepresenting the text feature sequence extracted by the BiGRU.
In the embodiment, the visual semantic feature extraction module takes the output result of the convolutional neural network as the low-level feature of the image, and fuses with the tampering detection part to realize the analysis of the image physical layer. To obtain a better semantic representation of the visual part, the input image is first encoded using the ResNet50 pre-training model. The image features are encoded using a 1024-dimensional fully-connected layer before the classification layer of the pre-trained ResNet50 model. The image representation is a 1024-dimensional vector. The process is shown in formula (3):
Figure BDA0003519652010000043
wherein v isiRepresenting the input original image, hv iRepresenting the visual semantic features extracted by the ResNet 50. The hierarchical structure of the input image is computed as a weighted sum of different feature vectors. During this training process, vectors are initialized randomly and learned together. Up to now, input data has been obtained. Semantic features of the image are then accepted through an attention mechanism, highlighting portions of the image with strong emotional expressions. Thus, when a visual modal representation is obtained, each feature is given a weight to indicate its "importance" in the modal representation.
As shown in formulas (4) to (6):
Figure BDA0003519652010000051
Figure BDA0003519652010000052
Figure BDA0003519652010000053
wherein WiRepresenting a weight matrix, biIs a deviation term, UTRepresenting a transposable weight vector, u is a scoring function that evaluates the importance of each feature vector. And then, carrying out normalization weighting on the ith feature vector i by utilizing a SoftMax function, and representing the high-level semantic meaning of the image. To better represent the images, the BiGRU is used to form the image ordering module. The image sequence module is typically used for the generation of image descriptions, aligning image features with text features. An image sequence module is introduced, and the feature expression of the image is sent to the BiGRU, so that the semantic vector of the visual mode is obtained. The step is equivalent to a commonly used embedding layer in text analysis, and the semantic feature analysis of the image is converted into the hierarchy of a semantic sequence.
Compared with directly utilizing the features of the image, the semantic information of the image is more favorably expressed, and the semantic information is expressed by the following formula (7):
Figure BDA0003519652010000054
in the present embodiment, the visual tamper feature extraction module performs information processing on an image by visual transformation or tampering, and then extracts an image tamper feature by applying the ResNet50 model, as shown in equation (8) to equation (9):
Figure BDA0003519652010000061
Figure BDA0003519652010000062
wherein v isiRepresenting the original image of the input. v. ofelaIs represented by EOriginal image of LA processing, hela iIndicating 245 tamper features extracted by ResNet 50.
In this embodiment, in the visual tampering feature extraction module, the visual semantic feature extraction module and the text feature extraction module obtain vector representations of texts and images. Proving that two sub-network modules learn the common representation space of image and text patterns, applying a fully connected layer at the last layer of each sub-network module and forcing the two sub-networks to share the weight of the last layer. Through feature sharing, semantic representation of images and texts is obtained. This is to visually present the same category of image and text examples. The cosine similarity is then used to measure the similarity between the image and the text. As shown in equation (10):
Figure BDA0003519652010000063
wherein s ist，svRespectively representing an image semantic sequence and a text feature sequence.
The value range of s is [ -1, 1], where-1 represents 265 the similarity of the text and the image 0, and 1 represents the similarity of the text and the image 1. To map the similarity between 0 and 1, an attempt is made to select the sigmoid activation function here to map the similarity between [0, 1], as shown by equation (11):
ps＝sigmoid(s) (11)
where sigmoid is the activation function used to map the similarity between 0 and 1.
Given the sheer similarity analysis, news articles formed by text and visual information mismatches are more susceptible to tampering than news articles formed by image and text matches. Next, a cross-entropy based loss function can be established, as follows:
τs＝-E(a，y)～(A，Y)(ylog(1-ps)+(1-y)logpa) (12)
Figure BDA0003519652010000071
in the present embodiment, in the multi-modal fusion module, the feature representation h of the image physical layer is obtained by the four sub-network modulesv,helaAnd a feature representation s of the image and text semantic layersc＝[sv,st]. In the multi-modal fusion module, the attention mechanism is utilized to assign the weighted image features and the semantic hierarchy features of the image and the text to the physical hierarchy, and the weighted image features and the semantic hierarchy features can be expressed as fc＝[sc,fp,hv,hela]. To highlight more valuable features, as shown in formulas (14) -16:
Figure BDA0003519652010000072
Figure BDA0003519652010000073
Figure BDA0003519652010000074
Wirepresenting a weight matrix, biIs a deviation term, UTRepresenting a transposable weight vector, u is a scoring function that weights the importance of each feature vector. Meanwhile, through the step, the fusion characteristics of the image and the text are obtained, and attention weight is given. The goal of (1) is to map text and visual features in the news onto tags, thereby predicting the probability that they are false news. The correspondence between features and tags is achieved by applying the Softmax function, as shown in equation (17)
pc＝softmax(Wp·se+bp) (17)
And defines a cross-entropy based loss function:
τp(θt,θv,θp)＝-E(a,y)～(A,Y)(y·logpc+(1-y)·logpc)
in order to better combine the similarity between pictures and texts, the semantic features of graphics and the visual physical hierarchy features, the final loss function is as follows (18):
τ(θt,θv,θp)＝ατp(θt,θv,θp)+βτs(θt,θv) (18)
wherein the parameters can be jointly learned by:
Figure BDA0003519652010000075
referring to fig. 3, a comparison graph of the before and after state processed by the compression or error level analysis algorithm, wherein (a) is the tampered image and (b) is the image processed by ELA, it can be seen that the hastelly is highlighted as the tampered portion. (c) Is an image that has not been recompressed and processed by ELA. (d) After (c) recompression transformation after ELA processing, it can be seen that the recompressed image and the original image after ELA transformation show different features. Therefore, the ELA algorithm can better highlight the malicious splicing and recompression characteristics of the false image.
The above description is intended to be illustrative of the preferred embodiment of the present invention and should not be taken as limiting the invention, but rather, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Claims (8)

1. A false news detection method based on a collaborative neural network is characterized in that: the method comprises a text feature extraction module, a visual semantic feature extraction module, a visual tampering feature extraction module, a similarity measurement module and a multi-mode fusion module, and further comprises the following steps:
step S1: the text feature extraction module and the visual semantic feature extraction module are responsible for extracting text and visual semantic features and mapping the text and the visual semantic features to the same space;
step S2: the visual tampering feature extraction module is responsible for extracting visual physical features and tampering features;
step S3: the similarity measurement module can directly measure the similarity of the news multi-mode data aiming at the problem that the image is not matched with the text.
2. The false news detection method based on the cooperative neural network as claimed in claim 1, wherein: in the text feature extraction module, a BERT pre-training model is adopted to extract text features, a BiGRU is used to extract the features extracted by the BERT, the BiGRU is used to extract the time attributes of the text features, and the text features are converted into a text feature sequence.
3. The false news detection method based on the cooperative neural network as claimed in claim 1, wherein: and in the visual semantic feature extraction module, the output of the convolutional neural network is used as the low-level feature of the image and then is fused with the tampering detection part.
4. A false news detection method based on a collaborative neural network as claimed in claim 3, characterized in that: the input image is encoded by a ResNet50 pre-trained model, and the image features are encoded using a 1024-dimensional fully connected layer before the classification layer of the pre-trained ResNet50 model.
5. The false news detection method based on the cooperative neural network as claimed in claim 1, wherein: in the visual tampering feature extraction module, the image is subjected to information processing through visual transformation or tampering, and an ResNet50 model is applied to extract image tampering features.
6. The false news detection method based on the cooperative neural network as claimed in claim 1, wherein: in the similarity measurement module, vector representation of texts and images is obtained through a visual semantic feature extraction module and a text feature extraction module.
7. The false news detection method based on the cooperative neural network as claimed in claim 1, wherein: in the multi-mode fusion module, fusion characteristics of the image and the text are obtained through a text characteristic extraction module, a visual semantic characteristic extraction module, a visual tampering characteristic extraction module and a similarity measurement module, and attention weight is given.
8. The false news detection method based on the cooperative neural network as claimed in claim 7, wherein: in the multi-mode fusion module, a weight image feature and semantic hierarchy features of images and texts are distributed to a physical hierarchy by using an attention mechanism.
CN202210173809.9A 2022-02-24 2022-02-24 False news detection method based on cooperative neural network Pending CN114662497A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210173809.9A CN114662497A (en) 2022-02-24 2022-02-24 False news detection method based on cooperative neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210173809.9A CN114662497A (en) 2022-02-24 2022-02-24 False news detection method based on cooperative neural network

Publications (1)

Publication Number Publication Date
CN114662497A true CN114662497A (en) 2022-06-24

Family

ID=82027011

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210173809.9A Pending CN114662497A (en) 2022-02-24 2022-02-24 False news detection method based on cooperative neural network

Country Status (1)

Country Link
CN (1) CN114662497A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115309860A (en) * 2022-07-18 2022-11-08 黑龙江大学 False news detection method based on pseudo twin network
CN115423050A (en) * 2022-11-04 2022-12-02 暨南大学 False news detection method and device, electronic equipment and storage medium
CN115809327A (en) * 2023-02-08 2023-03-17 四川大学 Real-time social network rumor detection method for multi-mode fusion and topics
CN117370679A (en) * 2023-12-06 2024-01-09 之江实验室 Method and device for verifying false messages of multi-mode bidirectional implication social network
CN117391051A (en) * 2023-12-12 2024-01-12 江西师范大学 Emotion-fused common attention network multi-modal false news detection method

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115309860A (en) * 2022-07-18 2022-11-08 黑龙江大学 False news detection method based on pseudo twin network
CN115423050A (en) * 2022-11-04 2022-12-02 暨南大学 False news detection method and device, electronic equipment and storage medium
CN115809327A (en) * 2023-02-08 2023-03-17 四川大学 Real-time social network rumor detection method for multi-mode fusion and topics
CN115809327B (en) * 2023-02-08 2023-05-05 四川大学 Real-time social network rumor detection method based on multimode fusion and topics
CN117370679A (en) * 2023-12-06 2024-01-09 之江实验室 Method and device for verifying false messages of multi-mode bidirectional implication social network
CN117370679B (en) * 2023-12-06 2024-03-26 之江实验室 Method and device for verifying false messages of multi-mode bidirectional implication social network
CN117391051A (en) * 2023-12-12 2024-01-12 江西师范大学 Emotion-fused common attention network multi-modal false news detection method
CN117391051B (en) * 2023-12-12 2024-03-08 江西师范大学 Emotion-fused common attention network multi-modal false news detection method

Similar Documents

Publication Publication Date Title
CN114662497A (en) False news detection method based on cooperative neural network
CN106022300B (en) Traffic sign recognition method and system based on cascade deep study
Guo et al. Human attribute recognition by refining attention heat map
CN111061843A (en) Knowledge graph guided false news detection method
CN112036276B (en) Artificial intelligent video question-answering method
CN104504362A (en) Face detection method based on convolutional neural network
CN113705218B (en) Event element gridding extraction method based on character embedding, storage medium and electronic device
Li et al. Image manipulation localization using attentional cross-domain CNN features
CN117521012A (en) False information detection method based on multi-mode context hierarchical step alignment
CN113469214A (en) False news detection method and device, electronic equipment and storage medium
CN116955707A (en) Content tag determination method, device, equipment, medium and program product
Yuan et al. Vsr++: Improving visual semantic reasoning for fine-grained image-text matching
CN117609765A (en) Multi-modal false news detection method
CN117149944A (en) Multi-mode situation emotion recognition method and system based on wide time range
CN118211141A (en) Social media false news detection method based on multi-modal fusion
CN114662586A (en) Method for detecting false information based on common attention multi-mode fusion mechanism
Liu et al. Iterative deep neighborhood: a deep learning model which involves both input data points and their neighbors
CN117235605B (en) Sensitive information classification method and device based on multi-mode attention fusion
CN110472655A (en) A kind of marker machine learning identifying system and method for border tourism
CN111859925B (en) Emotion analysis system and method based on probability emotion dictionary
CN116958642A (en) Picture classification method and device, electronic equipment and storage medium
CN115982652A (en) Cross-modal emotion analysis method based on attention network
CN113283535B (en) False message detection method and device integrating multi-mode characteristics
CN113297934A (en) Multi-mode video behavior analysis method for detecting internet violent harmful scene
CN116361785B (en) Recommendation system malicious attack detection method and system based on deep learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination